diff --git a/docs/en/engines/table-engines/integrations/iceberg.md b/docs/en/engines/table-engines/integrations/iceberg.md index e36e90ffb3fd..19b87ea5c1a9 100644 --- a/docs/en/engines/table-engines/integrations/iceberg.md +++ b/docs/en/engines/table-engines/integrations/iceberg.md @@ -131,9 +131,19 @@ ClickHouse supports reading Iceberg tables that use the following deletion metho - [Position deletes](https://iceberg.apache.org/spec/#position-delete-files) - [Equality deletes](https://iceberg.apache.org/spec/#equality-delete-files) (supported from version 25.8+) +- [Deletion vectors](https://iceberg.apache.org/spec/#deletion-vectors) stored in Puffin files (Iceberg v3, read-only) -The following deletion method is **not supported**: -- [Deletion vectors](https://iceberg.apache.org/spec/#deletion-vectors) (introduced in v3) +The following limitations apply to deletion vectors: + +- Only `deletion-vector-v1` Puffin blobs are supported +- Data files must be in Parquet format +- Column-scoped deletion vectors (user column ids in puffin `fields`) are not supported. Writers may set `fields` to `[]` or to the Iceberg reserved `_pos` id (`2147483645`) for file-scoped deletion vectors. +- Writing deletion vectors is not supported +- `DELETE` / `UPDATE` mutations on Iceberg format version 3+ tables are rejected (writers must not add position-delete files) + +Parsed deletion vectors can be cached in memory when `use_puffin_files_cache` is enabled and the puffin file has a non-empty `etag`. Empty deletion vectors are cached as well, so repeated reads do not re-fetch the puffin file. Parsed footers for coalesced multi-DV Puffin files are memoized with that cache (same identity: storage, path, `etag`) so slices share one footer parse; the memo shares `puffin_files_cache_size` / max-entry limits and is dropped when the cache is disabled (`puffin_files_cache_size=0`) or cleared. The cache can be cleared with `SYSTEM DROP PUFFIN FILES CACHE`. + +For [`icebergCluster`](/sql-reference/table-functions/icebergCluster.md) (and `object_storage_cluster`), the initiator loads and materializes each data file's deletion vector while distributing tasks, then sends the resulting row bitmap to workers with the task. Workers apply the bitmap; they do not re-read the Puffin blob for that path. On wide v3 tables this can make the initiator a serialization point for deletion-vector I/O and decode. ### Basic usage {#basic-usage} ```sql diff --git a/docs/en/sql-reference/statements/system.md b/docs/en/sql-reference/statements/system.md index 5d50202e6aca..f1d0b34be88e 100644 --- a/docs/en/sql-reference/statements/system.md +++ b/docs/en/sql-reference/statements/system.md @@ -109,6 +109,10 @@ Clears the per-URL Confluent Schema Registry caches used by the `AvroConfluent` Clears the parquet metadata cache. +## SYSTEM DROP PUFFIN FILES CACHE {#drop-puffin-files-cache} + +Clears the Puffin files cache used for parsed Iceberg puffin file content such as deletion vectors. + ## SYSTEM CLEAR|DROP TEXT INDEX CACHES {#drop-text-index-caches} Clears the text index's header, dictionary and postings caches. diff --git a/docs/en/sql-reference/table-functions/iceberg.md b/docs/en/sql-reference/table-functions/iceberg.md index 64b9d9c62ba6..1abff945e6bb 100644 --- a/docs/en/sql-reference/table-functions/iceberg.md +++ b/docs/en/sql-reference/table-functions/iceberg.md @@ -120,11 +120,23 @@ ClickHouse supports time travel for Iceberg tables, allowing you to query histor ## Processing of tables with deleted rows {#deleted-rows} -Currently, only Iceberg tables with [position deletes](https://iceberg.apache.org/spec/#position-delete-files) are supported. +ClickHouse supports reading Iceberg tables that use the following deletion methods: -The following deletion methods are **not supported**: -- [Equality deletes](https://iceberg.apache.org/spec/#equality-delete-files) -- [Deletion vectors](https://iceberg.apache.org/spec/#deletion-vectors) (introduced in v3) +- [Position deletes](https://iceberg.apache.org/spec/#position-delete-files) +- [Equality deletes](https://iceberg.apache.org/spec/#equality-delete-files) (supported from version 25.8+) +- [Deletion vectors](https://iceberg.apache.org/spec/#deletion-vectors) stored in Puffin files (Iceberg v3, read-only) + +The following limitations apply to deletion vectors: + +- Only `deletion-vector-v1` Puffin blobs are supported +- Data files must be in Parquet format +- Column-scoped deletion vectors (user column ids in puffin `fields`) are not supported. Writers may set `fields` to `[]` or to the Iceberg reserved `_pos` id (`2147483645`) for file-scoped deletion vectors. +- Writing deletion vectors is not supported +- `DELETE` / `UPDATE` mutations on Iceberg format version 3+ tables are rejected (writers must not add position-delete files) + +Parsed deletion vectors can be cached in memory when `use_puffin_files_cache` is enabled and the puffin file has a non-empty `etag`. Empty deletion vectors are cached as well, so repeated reads do not re-fetch the puffin file. Parsed footers for coalesced multi-DV Puffin files are memoized with that cache (same identity: storage, path, `etag`) so slices share one footer parse; the memo shares `puffin_files_cache_size` / max-entry limits and is dropped when the cache is disabled (`puffin_files_cache_size=0`) or cleared. The cache can be cleared with `SYSTEM DROP PUFFIN FILES CACHE`. + +For [`icebergCluster`](/sql-reference/table-functions/icebergCluster.md) (and `object_storage_cluster`), the initiator loads and materializes each data file's deletion vector while distributing tasks, then sends the resulting row bitmap to workers with the task. Workers apply the bitmap; they do not re-read the Puffin blob for that path. On wide v3 tables this can make the initiator a serialization point for deletion-vector I/O and decode. ### Basic usage {#basic-usage} @@ -384,8 +396,9 @@ y: 993 ### DELETE {#iceberg-writes-delete} -Deleting extra rows in the merge-on-read format is also supported in ClickHouse. +Deleting extra rows in the merge-on-read format is also supported in ClickHouse for Iceberg format version 2. This query will create a new snapshot with position delete files. +Mutations on format version 3+ tables are rejected until ClickHouse can write deletion vectors (Iceberg v3 writers must not add new position-delete files). ### Example {#example-iceberg-writes-delete} diff --git a/docs/en/sql-reference/table-functions/icebergCluster.md b/docs/en/sql-reference/table-functions/icebergCluster.md index 91d6c9dea2ac..312e5b30ec0b 100644 --- a/docs/en/sql-reference/table-functions/icebergCluster.md +++ b/docs/en/sql-reference/table-functions/icebergCluster.md @@ -12,6 +12,12 @@ This is an extension to the [iceberg](/sql-reference/table-functions/iceberg.md) Allows processing files from Apache [Iceberg](https://iceberg.apache.org/) in parallel from many nodes in a specified cluster. On initiator it creates a connection to all nodes in the cluster and dispatches each file dynamically. On the worker node it asks the initiator about the next task to process and processes it. This is repeated until all tasks are finished. +## Deletion vectors on cluster reads {#deletion-vectors-cluster} + +Iceberg v3 [deletion vectors](https://iceberg.apache.org/spec/#deletion-vectors) are loaded on the **initiator** while it distributes tasks: for each data file the initiator reads the Puffin blob, validates it, materializes deleted row positions, and attaches the bitmap to the task sent to workers. Workers apply that bitmap when reading Parquet; they do not fetch or parse the Puffin file again for that path. See [Processing of tables with deleted rows](/sql-reference/table-functions/iceberg.md#deleted-rows) for format limits and caching. + +On wide tables with many deletion vectors, initiator-side decode and per-task bitmap serialization can become a bottleneck even when Parquet reads are well parallelized across the cluster. + ## Syntax {#syntax} ```sql diff --git a/docs/reference/formats/Puffin/Puffin.mdx b/docs/reference/formats/Puffin/Puffin.mdx index 96f321ad4d0a..81b5764ad215 100644 --- a/docs/reference/formats/Puffin/Puffin.mdx +++ b/docs/reference/formats/Puffin/Puffin.mdx @@ -1,6 +1,10 @@ --- description: 'Documentation for the Puffin format' +input_format: true +output_format: false keywords: ['Puffin'] +sidebar_label: 'Puffin' +sidebar_position: 1 slug: /interfaces/formats/Puffin title: 'Puffin' doc_type: 'reference' @@ -10,16 +14,16 @@ doc_type: 'reference' Input format for reading [Apache Iceberg Puffin](https://iceberg.apache.org/puffin-spec/) files. -The format exposes deleted row positions from `deletion-vector-v1` blobs. It is the only supported blob type: a file containing any other blob type (for example `apache-datasketches-theta-v1`) is rejected. +The format exposes deleted row positions from `deletion-vector-v1` blobs. Other blob types (for example `apache-datasketches-theta-v1`) are skipped. If a puffin file contains multiple `deletion-vector-v1` blobs, the format outputs one row per such blob. Fixed output columns: - `referenced_data_file` (`String`) - location of the data file the deletion vector applies to (`referenced-data-file` blob property) - `deleted_rows` (`Array(UInt64)`) - 64-bit row positions deleted according to the deletion vector roaring bitmap -Deletion vectors whose declared `cardinality` exceeds an absolute materialization ceiling are rejected when `deleted_rows` is requested. Footer `deletion-vector-v1` properties (including that `cardinality` parses as an unsigned integer) are always validated. Selecting only `referenced_data_file` skips on-disk payload I/O and therefore also skips envelope, CRC, roaring deserialize, and the materialization ceiling — intentionally, so a path-only projection does not read up to the blob-size cap. +Deletion vectors whose declared `cardinality` exceeds an absolute materialization ceiling are rejected when `deleted_rows` is requested, **before** envelope peek or full blob allocation (same fail-closed order as the Iceberg deletion-vector reader). Footer `deletion-vector-v1` properties are always validated: `cardinality` must parse as an unsigned integer, `snapshot-id` / `sequence-number` must be `-1`, and `fields` must be either empty or the singleton Iceberg reserved `_pos` id (`2147483645`) that Spark writes for file-scoped DVs — other `fields` lists (column-scoped DVs) are rejected. Selecting only `referenced_data_file` skips on-disk payload I/O and therefore also skips envelope, CRC, roaring deserialize, and the materialization ceiling — intentionally, so a path-only projection does not read up to the blob-size cap. -On-disk `deletion-vector-v1` blob length is bounded by an absolute ceiling (aligned with Iceberg's 2 GiB content-size check). When `deleted_rows` is requested, the reader peeks the envelope header (combined length and magic) before allocating the full payload; CRC is verified after the bounded read. +On-disk `deletion-vector-v1` blob length is bounded by an absolute ceiling (aligned with Iceberg's 2 GiB content-size check). When `deleted_rows` is requested and cardinality is within the materialization ceiling, the reader peeks the envelope header (combined length and magic) before allocating the full payload; CRC is verified after the bounded read. LZ4-compressed and uncompressed puffin footers are supported. Footer payload size (and declared LZ4 content size) is bounded by a compression ratio where applicable and an absolute ceiling; oversized footers are rejected before allocation. diff --git a/docs/reference/formats/Puffin/PuffinMetadata.mdx b/docs/reference/formats/Puffin/PuffinMetadata.mdx index cace329a9705..7c45ac36f764 100644 --- a/docs/reference/formats/Puffin/PuffinMetadata.mdx +++ b/docs/reference/formats/Puffin/PuffinMetadata.mdx @@ -1,6 +1,10 @@ --- description: 'Documentation for the PuffinMetadata format' +input_format: true +output_format: false keywords: ['PuffinMetadata'] +sidebar_label: 'PuffinMetadata' +sidebar_position: 2 slug: /interfaces/formats/PuffinMetadata title: 'PuffinMetadata' doc_type: 'reference' @@ -9,9 +13,7 @@ doc_type: 'reference' ## Description {#description} Special input format for reading [Apache Iceberg Puffin](https://iceberg.apache.org/puffin-spec/) file footer metadata. -It outputs one row per blob entry from the footer `BlobMetadata` list. - -`deletion-vector-v1` is the only supported blob type: a file containing any other blob type (for example `apache-datasketches-theta-v1`) is rejected. +It outputs one row per blob entry from the footer `BlobMetadata` list, including non-deletion-vector types (for example `apache-datasketches-theta-v1`). Full deletion-vector property validation applies only to `deletion-vector-v1` entries. Fixed output columns: - `blob_type` (`String`) - blob type, for example `deletion-vector-v1` diff --git a/programs/local/LocalServer.cpp b/programs/local/LocalServer.cpp index efaa6c71a527..1e8367e6d0c3 100644 --- a/programs/local/LocalServer.cpp +++ b/programs/local/LocalServer.cpp @@ -174,6 +174,10 @@ namespace ServerSetting extern const ServerSettingsUInt64 parquet_metadata_cache_size; extern const ServerSettingsUInt64 parquet_metadata_cache_max_entries; extern const ServerSettingsDouble parquet_metadata_cache_size_ratio; + extern const ServerSettingsString puffin_files_cache_policy; + extern const ServerSettingsUInt64 puffin_files_cache_size; + extern const ServerSettingsUInt64 puffin_files_cache_max_entries; + extern const ServerSettingsDouble puffin_files_cache_size_ratio; extern const ServerSettingsUInt64 max_active_parts_loading_thread_pool_size; extern const ServerSettingsUInt64 max_io_thread_pool_free_size; extern const ServerSettingsUInt64 max_io_thread_pool_size; @@ -1546,6 +1550,17 @@ void LocalServer::processConfig() global_context->setParquetMetadataCache(parquet_metadata_cache_policy, parquet_metadata_cache_size, parquet_metadata_cache_max_entries, parquet_metadata_cache_size_ratio); #endif + String puffin_files_cache_policy = server_settings[ServerSetting::puffin_files_cache_policy]; + size_t puffin_files_cache_size = server_settings[ServerSetting::puffin_files_cache_size]; + size_t puffin_files_cache_max_entries = server_settings[ServerSetting::puffin_files_cache_max_entries]; + double puffin_files_cache_size_ratio = server_settings[ServerSetting::puffin_files_cache_size_ratio]; + if (puffin_files_cache_size > max_cache_size) + { + puffin_files_cache_size = max_cache_size; + LOG_INFO(log, "Lowered Puffin files cache size to {} because the system has limited RAM", formatReadableSizeWithBinarySuffix(puffin_files_cache_size)); + } + global_context->setPuffinFilesCache(puffin_files_cache_policy, puffin_files_cache_size, puffin_files_cache_max_entries, puffin_files_cache_size_ratio); + Names allowed_disks_table_engines; splitInto<','>(allowed_disks_table_engines, server_settings[ServerSetting::allowed_disks_for_table_engines].value); global_context->setAllowedDisksForTableEngines(std::unordered_set(allowed_disks_table_engines.begin(), allowed_disks_table_engines.end())); diff --git a/programs/server/Server.cpp b/programs/server/Server.cpp index ccd977bc4de0..97e580b77d63 100644 --- a/programs/server/Server.cpp +++ b/programs/server/Server.cpp @@ -290,6 +290,10 @@ namespace ServerSetting extern const ServerSettingsUInt64 parquet_metadata_cache_size; extern const ServerSettingsUInt64 parquet_metadata_cache_max_entries; extern const ServerSettingsDouble parquet_metadata_cache_size_ratio; + extern const ServerSettingsString puffin_files_cache_policy; + extern const ServerSettingsUInt64 puffin_files_cache_size; + extern const ServerSettingsUInt64 puffin_files_cache_max_entries; + extern const ServerSettingsDouble puffin_files_cache_size_ratio; extern const ServerSettingsUInt64 io_thread_pool_queue_size; extern const ServerSettingsBool jemalloc_enable_global_profiler; extern const ServerSettingsBool jemalloc_collect_global_profile_samples_in_trace_log; @@ -2302,6 +2306,16 @@ try } global_context->setParquetMetadataCache(parquet_metadata_cache_policy, parquet_metadata_cache_size, parquet_metadata_cache_max_entries, parquet_metadata_cache_size_ratio); #endif + String puffin_files_cache_policy = server_settings[ServerSetting::puffin_files_cache_policy]; + size_t puffin_files_cache_size = server_settings[ServerSetting::puffin_files_cache_size]; + size_t puffin_files_cache_max_entries = server_settings[ServerSetting::puffin_files_cache_max_entries]; + double puffin_files_cache_size_ratio = server_settings[ServerSetting::puffin_files_cache_size_ratio]; + if (puffin_files_cache_size > max_cache_size) + { + puffin_files_cache_size = max_cache_size; + LOG_INFO(log, "Lowered Puffin files cache size to {} because the system has limited RAM", formatReadableSizeWithBinarySuffix(puffin_files_cache_size)); + } + global_context->setPuffinFilesCache(puffin_files_cache_policy, puffin_files_cache_size, puffin_files_cache_max_entries, puffin_files_cache_size_ratio); Names allowed_disks_table_engines; splitInto<','>(allowed_disks_table_engines, server_settings[ServerSetting::allowed_disks_for_table_engines].value); @@ -2731,6 +2745,7 @@ try #if USE_PARQUET global_context->updateParquetMetadataCacheConfiguration(config(), max_cache_size_in_bytes); #endif + global_context->updatePuffinFilesCacheConfiguration(config(), max_cache_size_in_bytes); } #if USE_SSL diff --git a/src/Access/Common/AccessType.h b/src/Access/Common/AccessType.h index 97edf6a18160..f32f934c224b 100644 --- a/src/Access/Common/AccessType.h +++ b/src/Access/Common/AccessType.h @@ -320,6 +320,7 @@ enum class AccessType : uint8_t M(SYSTEM_DROP_ICEBERG_METADATA_CACHE, "SYSTEM CLEAR ICEBERG_METADATA_CACHE, SYSTEM DROP ICEBERG_METADATA_CACHE", GLOBAL, SYSTEM_DROP_CACHE) \ M(SYSTEM_DROP_AVRO_SCHEMA_CACHE, "SYSTEM CLEAR AVRO SCHEMA CACHE, SYSTEM DROP AVRO SCHEMA CACHE, DROP AVRO SCHEMA CACHE", GLOBAL, SYSTEM_DROP_CACHE) \ M(SYSTEM_DROP_PARQUET_METADATA_CACHE, "SYSTEM DROP PARQUET_METADATA_CACHE", GLOBAL, SYSTEM_DROP_CACHE) \ + M(SYSTEM_DROP_PUFFIN_FILES_CACHE, "SYSTEM DROP PUFFIN_FILES_CACHE", GLOBAL, SYSTEM_DROP_CACHE) \ M(SYSTEM_PREWARM_PRIMARY_INDEX_CACHE, "SYSTEM PREWARM PRIMARY INDEX, PREWARM PRIMARY INDEX CACHE, PREWARM PRIMARY INDEX", GLOBAL, SYSTEM_DROP_CACHE) \ M(SYSTEM_DROP_PRIMARY_INDEX_CACHE, "SYSTEM CLEAR PRIMARY INDEX CACHE, SYSTEM DROP PRIMARY INDEX, DROP PRIMARY INDEX CACHE, DROP PRIMARY INDEX", GLOBAL, SYSTEM_DROP_CACHE) \ M(SYSTEM_DROP_UNCOMPRESSED_CACHE, "SYSTEM CLEAR UNCOMPRESSED CACHE, SYSTEM DROP UNCOMPRESSED, DROP UNCOMPRESSED CACHE, DROP UNCOMPRESSED", GLOBAL, SYSTEM_DROP_CACHE) \ diff --git a/src/AggregateFunctions/AggregateFunctionGroupBitmapData.h b/src/AggregateFunctions/AggregateFunctionGroupBitmapData.h index fde3743850ba..7fa193574639 100644 --- a/src/AggregateFunctions/AggregateFunctionGroupBitmapData.h +++ b/src/AggregateFunctions/AggregateFunctionGroupBitmapData.h @@ -34,6 +34,72 @@ enum BitmapKind Bitmap = 1 }; +/// Approximate heap footprint of a 32-bit CRoaring bitmap (index capacity + container capacities). +/// Unlike `Roaring::getSizeInBytes()`, array/run containers use allocated capacity, not cardinality. +inline UInt64 estimateRoaring32AllocatedBytes(const roaring::Roaring & bitmap) +{ + using namespace roaring::internal; + const roaring::roaring_array_t * ra = &bitmap.roaring.high_low_container; + UInt64 bytes = sizeof(roaring::Roaring); + if (ra->allocation_size > 0) + { + bytes += static_cast(ra->allocation_size) + * (sizeof(uint16_t) + sizeof(uint8_t) + sizeof(container_t *)); + } + for (int32_t i = 0; i < ra->size; ++i) + { + uint8_t typecode = ra->typecodes[i]; + const container_t * c = container_unwrap_shared(ra->containers[i], &typecode); + switch (typecode) + { + case ARRAY_CONTAINER_TYPE: + { + const array_container_t * ac = const_CAST_array(c); + bytes += sizeof(array_container_t) + static_cast(ac->capacity) * sizeof(uint16_t); + break; + } + case BITSET_CONTAINER_TYPE: + { + bytes += sizeof(bitset_container_t) + BITSET_CONTAINER_SIZE_IN_WORDS * sizeof(uint64_t); + break; + } + case RUN_CONTAINER_TYPE: + { + const run_container_t * rc = const_CAST_run(c); + bytes += sizeof(run_container_t) + static_cast(rc->capacity) * sizeof(rle16_t); + break; + } + default: + break; + } + } + return bytes; +} + +/// Count distinct high-32 keys in a Roaring64Map. Fast-path when min/max share one key. +inline UInt64 countRoaring64MapHighKeys(const roaring::Roaring64Map & bitmap) +{ + if (bitmap.isEmpty()) + return 0; + + const UInt64 min_value = bitmap.minimum(); + const UInt64 max_value = bitmap.maximum(); + if ((min_value >> 32) == (max_value >> 32)) + return 1; + + UInt64 keys = 0; + UInt64 prev_high = ~UInt64{0}; + for (auto it = bitmap.begin(); it != bitmap.end(); ++it) + { + const UInt64 high = static_cast(*it) >> 32; + if (high != prev_high) + { + ++keys; + prev_high = high; + } + } + return keys; +} /** * For a small number of values - an array of fixed size "on the stack". @@ -93,6 +159,34 @@ class RoaringBitmapWithSmallSet : private boost::noncopyable return roaring_bitmap->cardinality(); } + UInt64 getAllocatedBytes() const + { + if (isSmall()) + return sizeof(small); + + /// Prefer a heap estimate over `getSizeInBytes()` (serialization size). Include a small + /// allowance for the shared_ptr control block that owns `roaring_bitmap`. + constexpr UInt64 SHARED_PTR_CONTROL_BLOCK = 32; + + if constexpr (sizeof(T) < 8) + { + return estimateRoaring32AllocatedBytes(*roaring_bitmap) + SHARED_PTR_CONTROL_BLOCK; + } + else + { + /// Roaring64Map keeps roarings private; approximate as native serialization size plus + /// per-high-key map/Roaring/container overhead (serialization undercounts capacity and + /// std::map nodes — important for sparse high keys). + constexpr UInt64 PER_HIGH_KEY_OVERHEAD = + 4 * sizeof(void *) + sizeof(UInt32) + sizeof(roaring::Roaring) + 64; + + const UInt64 serialized = roaring_bitmap->getSizeInBytes(/*portable=*/false); + const UInt64 high_keys = countRoaring64MapHighKeys(*roaring_bitmap); + return serialized + high_keys * PER_HIGH_KEY_OVERHEAD + sizeof(roaring::Roaring64Map) + + SHARED_PTR_CONTROL_BLOCK; + } + } + void merge(const RoaringBitmapWithSmallSet & r1) { if (r1.isLarge()) @@ -535,6 +629,51 @@ class RoaringBitmapWithSmallSet : private boost::noncopyable return count; } + /** + * Count set bits in `[range_start, range_end)` without allocating a result bitmap. + * Used by need-only-count DV filtering to avoid an O(N) dense Filter over file rows. + * Implemented via roaring `rank` so repeated per-row-group queries stay O(containers), + * not O(row_groups × cardinality). + */ + UInt64 rb_range_cardinality(UInt64 range_start, UInt64 range_end) const /// NOLINT + { + if (range_start >= range_end) + return 0; + + if (isSmall()) + { + UInt64 count = 0; + for (const auto & x : small) + { + const UInt64 val = static_cast(x.getValue()); + if (val >= range_start && val < range_end) + ++count; + } + return count; + } + + /// |bitmap ∩ [start, end)| = rank(end - 1) - rank(start - 1). Same formula as DeleteBitmap. + if constexpr (sizeof(T) < 8) + { + constexpr UInt64 max_row = std::numeric_limits::max(); + if (range_start > max_row) + return 0; + const UInt64 hi_inclusive = std::min(range_end - 1, max_row); + if (hi_inclusive < range_start) + return 0; + const UInt64 upper = roaring_bitmap->rank(static_cast(hi_inclusive)); + const UInt64 lower = (range_start == 0) ? 0 : roaring_bitmap->rank(static_cast(range_start - 1)); + return upper - lower; + } + else + { + const UInt64 hi_inclusive = range_end - 1; + const UInt64 upper = roaring_bitmap->rank(hi_inclusive); + const UInt64 lower = (range_start == 0) ? 0 : roaring_bitmap->rank(range_start - 1); + return upper - lower; + } + } + /** * Return new set of the smallest `limit` values in set which is no less than `range_start`. * It's used in subset and currently only support UInt32 diff --git a/src/Client/BuzzHouse/Generator/SessionSettings.cpp b/src/Client/BuzzHouse/Generator/SessionSettings.cpp index b4d56f78d57d..cdc597a90780 100644 --- a/src/Client/BuzzHouse/Generator/SessionSettings.cpp +++ b/src/Client/BuzzHouse/Generator/SessionSettings.cpp @@ -1645,6 +1645,7 @@ static std::unordered_map serverSettings2 = { {"use_page_cache_for_local_disks", trueOrFalseSetting}, {"use_page_cache_for_object_storage", trueOrFalseSetting}, {"use_parquet_metadata_cache", trueOrFalseSetting}, + {"use_puffin_files_cache", trueOrFalseSetting}, {"use_query_cache", trueOrFalseSetting}, {"use_roaring_bitmap_iceberg_positional_deletes", trueOrFalseSetting}, {"use_skip_indexes_if_final_exact_mode", CHSetting(trueOrFalse, {"0", "1"}, true)}, diff --git a/src/Common/CacheBase.h b/src/Common/CacheBase.h index 3ca63aa79488..29d06f6382bc 100644 --- a/src/Common/CacheBase.h +++ b/src/Common/CacheBase.h @@ -10,6 +10,7 @@ #include #include +#include #include #include #include @@ -23,6 +24,19 @@ namespace ErrorCodes extern const int BAD_ARGUMENTS; } +/// Outcome of `CacheBase::getOrSetWithOutcome`, decided under the cache lock together with the +/// returned value so callers can classify hits/misses without a follow-up `contains()` race. +enum class CacheGetOrSetOutcome : uint8_t +{ + /// Value was resident when returned (this caller did not run `load_func`). + Hit, + /// This caller ran `load_func` and inserted the value into the cache. + MissInserted, + /// Value was produced (by this caller or a stampede peer) but is not resident — e.g. a concurrent + /// `clear()` discarded the insert token. The returned `MappedPtr` is still valid for the caller. + MissNotResident, +}; + /// Thread-safe cache that evicts entries using special cache policy /// (default policy evicts entries which are not used for a long time). /// WeightFunction is a functor that takes Mapped as a parameter and returns "weight" (approximate size) @@ -156,9 +170,9 @@ class CacheBase /// Exceptions occurring in load_func will be propagated to the caller. Another thread from the /// set of concurrent threads will then try to call its load_func etc. /// - /// Returns std::pair of the cached value and a bool indicating whether the value was produced during this call. + /// Returns the value together with an outcome that is atomic with residency (see CacheGetOrSetOutcome). template - std::pair getOrSet(const Key & key, LoadFunc && load_func) + std::pair getOrSetWithOutcome(const Key & key, LoadFunc && load_func) { InsertTokenHolder token_holder; { @@ -167,7 +181,7 @@ class CacheBase if (val) { ++hits; - return std::make_pair(val, false); + return std::make_pair(val, CacheGetOrSetOutcome::Hit); } auto & token = insert_tokens[key]; @@ -186,8 +200,18 @@ class CacheBase if (token->value) { /// Another thread already produced the value while we waited for token->mutex. - ++hits; - return std::make_pair(token->value, false); + /// If a concurrent clear() discarded that insert, the value is not resident — + /// count a miss (same class of outcome as the producer when insertion is skipped). + { + std::lock_guard cache_lock(mutex); + if (auto cached = cache_policy->get(key)) + { + ++hits; + return std::make_pair(std::move(cached), CacheGetOrSetOutcome::Hit); + } + } + ++misses; + return std::make_pair(token->value, CacheGetOrSetOutcome::MissNotResident); } ++misses; @@ -197,18 +221,28 @@ class CacheBase /// Insert the new value only if the token is still in present in insert_tokens. /// (The token may be absent because of a concurrent clear() call). - bool result = false; auto token_it = insert_tokens.find(key); if (token_it != insert_tokens.end() && token_it->second.get() == token) { cache_policy->set(key, token->value); - result = true; + if (!token->cleaned_up) + token_holder.cleanup(token_lock, cache_lock); + return std::make_pair(token->value, CacheGetOrSetOutcome::MissInserted); } if (!token->cleaned_up) token_holder.cleanup(token_lock, cache_lock); - return std::make_pair(token->value, result); + return std::make_pair(token->value, CacheGetOrSetOutcome::MissNotResident); + } + + /// Same as getOrSetWithOutcome, but the bool is true only when this call inserted the value + /// (`CacheGetOrSetOutcome::MissInserted`). Prefer getOrSetWithOutcome when classifying hits/misses. + template + std::pair getOrSet(const Key & key, LoadFunc && load_func) + { + auto [value, outcome] = getOrSetWithOutcome(key, std::forward(load_func)); + return std::make_pair(std::move(value), outcome == CacheGetOrSetOutcome::MissInserted); } void getStats(size_t & out_hits, size_t & out_misses) const @@ -218,6 +252,17 @@ class CacheBase out_misses = misses; } + /// Number of concurrent getOrSet callers holding the insert token for `key`, or 0 if none. + /// Useful for tests that wait until a second thread has joined an in-flight load. + size_t getInsertTokenRefcount(const Key & key) const + { + std::lock_guard lock(mutex); + auto it = insert_tokens.find(key); + if (it == insert_tokens.end()) + return 0; + return it->second->refcount; + } + std::vector dump() const { std::lock_guard lock(mutex); diff --git a/src/Common/CurrentMetrics.cpp b/src/Common/CurrentMetrics.cpp index d479717611ad..45549016a56b 100644 --- a/src/Common/CurrentMetrics.cpp +++ b/src/Common/CurrentMetrics.cpp @@ -338,6 +338,8 @@ M(IcebergMetadataFilesCacheFiles, "Number of cached files in the Iceberg metadata cache") \ M(ParquetMetadataCacheBytes, "Size of the Parquet metadata cache in bytes") \ M(ParquetMetadataCacheFiles, "Number of cached files in the Parquet metadata cache") \ + M(PuffinFilesCacheBytes, "Size of the Puffin files cache in bytes") \ + M(PuffinFilesCacheFiles, "Number of cached entries in the Puffin files cache") \ M(AvroSchemaCacheBytes, "Size of the Avro schema cache in bytes") \ M(AvroSchemaCacheCells, "Number of cached Avro schemas, including both registered and fetched schemas.") \ M(AvroSchemaRegistryCacheBytes, "Size of the Avro schema registry cache in bytes") \ diff --git a/src/Common/ProfileEvents.cpp b/src/Common/ProfileEvents.cpp index e631970e61fe..84eb84a6accc 100644 --- a/src/Common/ProfileEvents.cpp +++ b/src/Common/ProfileEvents.cpp @@ -114,6 +114,11 @@ M(IcebergMetadataReturnedObjectInfos, "Total number of returned object infos from iceberg iterator.", ValueType::Number) \ M(IcebergMinMaxNonPrunedDeleteFiles, "Total number of accepted data files-position delete file pairs by minmax analysis from pairs suitable by partitioning and sequence number.", ValueType::Number) \ M(IcebergMinMaxPrunedDeleteFiles, "Total number of accepted data files-position delete file pairs by minmax analysis from pairs suitable by partitioning and sequence number.", ValueType::Number) \ + M(PuffinFilesRead, "Number of Puffin files read (footer or deletion vector blob).", ValueType::Number) \ + M(PuffinFileReadMicroseconds, "Total time spent reading Puffin files.", ValueType::Microseconds) \ + M(PuffinFilesCacheHits, "Number of times parsed Puffin file content has been found in the cache.", ValueType::Number) \ + M(PuffinFilesCacheMisses, "Number of times parsed Puffin file content has not been found in the cache and had to be read from disk.", ValueType::Number) \ + M(PuffinFilesCacheWeightLost, "Approximate number of bytes evicted from the Puffin files cache.", ValueType::Number) \ M(VectorSimilarityIndexCacheHits, "Number of times an index granule has been found in the vector index cache.", ValueType::Number) \ M(VectorSimilarityIndexCacheMisses, "Number of times an index granule has not been found in the vector index cache and had to be read from disk.", ValueType::Number) \ M(VectorSimilarityIndexCacheWeightLost, "Approximate number of bytes evicted from the vector index cache.", ValueType::Number) \ diff --git a/src/Common/tests/gtest_lru_cache.cpp b/src/Common/tests/gtest_lru_cache.cpp index ed9c3c34c45f..ff6ddf4df52c 100644 --- a/src/Common/tests/gtest_lru_cache.cpp +++ b/src/Common/tests/gtest_lru_cache.cpp @@ -105,6 +105,73 @@ TEST(LRUCache, getOrSet) ASSERT_TRUE(*value == 10); } +TEST(LRUCache, getOrSetWithOutcomeHitAndMissInserted) +{ + using SimpleCacheBase = DB::CacheBase; + SimpleCacheBase cache("LRU", CurrentMetrics::end(), CurrentMetrics::end(), /*max_size_in_bytes*/ 10, /*max_count*/ 10, /*size_ratio*/ 0.5); + + size_t loads = 0; + auto load = [&]() + { + ++loads; + return std::make_shared(42); + }; + + { + auto [value, outcome] = cache.getOrSetWithOutcome(1, load); + ASSERT_NE(value, nullptr); + EXPECT_EQ(*value, 42); + EXPECT_EQ(outcome, DB::CacheGetOrSetOutcome::MissInserted); + EXPECT_EQ(loads, 1u); + } + { + auto [value, outcome] = cache.getOrSetWithOutcome(1, load); + ASSERT_NE(value, nullptr); + EXPECT_EQ(*value, 42); + EXPECT_EQ(outcome, DB::CacheGetOrSetOutcome::Hit); + EXPECT_EQ(loads, 1u); + } +} + +TEST(LRUCache, getOrSetWithOutcomeClearDuringLoadIsMissNotResident) +{ + using SimpleCacheBase = DB::CacheBase; + SimpleCacheBase cache("LRU", CurrentMetrics::end(), CurrentMetrics::end(), /*max_size_in_bytes*/ 10, /*max_count*/ 10, /*size_ratio*/ 0.5); + + auto [value, outcome] = cache.getOrSetWithOutcome( + 1, + [&]() + { + cache.clear(); + return std::make_shared(7); + }); + + ASSERT_NE(value, nullptr); + EXPECT_EQ(*value, 7); + EXPECT_EQ(outcome, DB::CacheGetOrSetOutcome::MissNotResident); + EXPECT_FALSE(cache.contains(1)); +} + +TEST(LRUCache, getOrSetWithOutcomeHitRemainsHitIfClearedAfterReturn) +{ + using SimpleCacheBase = DB::CacheBase; + SimpleCacheBase cache("LRU", CurrentMetrics::end(), CurrentMetrics::end(), /*max_size_in_bytes*/ 10, /*max_count*/ 10, /*size_ratio*/ 0.5); + + cache.getOrSetWithOutcome(1, []() { return std::make_shared(1); }); + + auto [value, outcome] = cache.getOrSetWithOutcome(1, []() { return std::make_shared(2); }); + EXPECT_EQ(outcome, DB::CacheGetOrSetOutcome::Hit); + ASSERT_NE(value, nullptr); + EXPECT_EQ(*value, 1); + + /// A follow-up contains() after clear would return false; callers must use `outcome`, not a + /// second residency check, when classifying hits (PuffinFilesCache used to race here). + cache.clear(); + EXPECT_FALSE(cache.contains(1)); + EXPECT_EQ(outcome, DB::CacheGetOrSetOutcome::Hit); + EXPECT_EQ(*value, 1); +} + TEST(LRUCache, noOnRemoveEntryCallback) { diff --git a/src/Core/Defines.h b/src/Core/Defines.h index c99199f76446..b351507566e8 100644 --- a/src/Core/Defines.h +++ b/src/Core/Defines.h @@ -124,6 +124,10 @@ static constexpr auto DEFAULT_PARQUET_METADATA_CACHE_POLICY = "SLRU"; static constexpr auto DEFAULT_PARQUET_METADATA_CACHE_MAX_SIZE = 512_MiB; static constexpr auto DEFAULT_PARQUET_METADATA_CACHE_SIZE_RATIO = 0.5; static constexpr auto DEFAULT_PARQUET_METADATA_CACHE_MAX_ENTRIES = 5000; +static constexpr auto DEFAULT_PUFFIN_FILES_CACHE_POLICY = "SLRU"; +static constexpr auto DEFAULT_PUFFIN_FILES_CACHE_MAX_SIZE = 512_MiB; +static constexpr auto DEFAULT_PUFFIN_FILES_CACHE_SIZE_RATIO = 0.5; +static constexpr auto DEFAULT_PUFFIN_FILES_CACHE_MAX_ENTRIES = 5000; static constexpr auto DEFAULT_QUERY_CONDITION_CACHE_POLICY = "SLRU"; static constexpr auto DEFAULT_QUERY_CONDITION_CACHE_MAX_SIZE = 100_MiB; static constexpr auto DEFAULT_QUERY_CONDITION_CACHE_SIZE_RATIO = 0.5l; diff --git a/src/Core/ServerSettings.cpp b/src/Core/ServerSettings.cpp index 607909800f96..6e45d40bcd33 100644 --- a/src/Core/ServerSettings.cpp +++ b/src/Core/ServerSettings.cpp @@ -24,6 +24,7 @@ #if USE_PARQUET # include #endif +#include #include #include #include @@ -561,6 +562,10 @@ namespace DECLARE(UInt64, parquet_metadata_cache_size, DEFAULT_PARQUET_METADATA_CACHE_MAX_SIZE, "Maximum size of parquet metadata cache in bytes. Zero means disabled.", 0) \ DECLARE(UInt64, parquet_metadata_cache_max_entries, DEFAULT_PARQUET_METADATA_CACHE_MAX_ENTRIES, "Maximum size of parquet metadata files cache in entries. Zero means disabled.", 0) \ DECLARE(Double, parquet_metadata_cache_size_ratio, DEFAULT_PARQUET_METADATA_CACHE_SIZE_RATIO, "The size of the protected queue (in case of SLRU policy) in the parquet metadata cache relative to the cache's total size.", 0) \ + DECLARE(String, puffin_files_cache_policy, DEFAULT_PUFFIN_FILES_CACHE_POLICY, "Puffin files cache policy name (SLRU or LRU).", 0) \ + DECLARE(UInt64, puffin_files_cache_size, DEFAULT_PUFFIN_FILES_CACHE_MAX_SIZE, "Maximum size of Puffin files cache in bytes. Zero means disabled.", 0) \ + DECLARE(UInt64, puffin_files_cache_max_entries, DEFAULT_PUFFIN_FILES_CACHE_MAX_ENTRIES, "Maximum number of entries in the Puffin files cache. Zero means unlimited.", 0) \ + DECLARE(Double, puffin_files_cache_size_ratio, DEFAULT_PUFFIN_FILES_CACHE_SIZE_RATIO, "The size of the protected queue (in case of SLRU policy) in the Puffin files cache relative to the cache's total size.", 0) \ DECLARE(String, allowed_disks_for_table_engines, "", "List of disks allowed for use with Iceberg", 0) \ DECLARE(String, vector_similarity_index_cache_policy, DEFAULT_VECTOR_SIMILARITY_INDEX_CACHE_POLICY, "Vector similarity index cache policy name.", 0) \ DECLARE(UInt64, vector_similarity_index_cache_size, DEFAULT_VECTOR_SIMILARITY_INDEX_CACHE_MAX_SIZE, R"(Size of cache for vector similarity indexes. Zero means disabled. @@ -2034,6 +2039,10 @@ ChangeableSettingsMap collectChangeableServerSettings(ContextPtr context) {"parquet_metadata_cache_size", {std::to_string(context->getParquetMetadataCache()->maxSizeInBytes()), ChangeableWithoutRestart::Yes}}); #endif + if (context->getPuffinFilesCache()) + changeable_settings.insert( + {"puffin_files_cache_size", + {std::to_string(context->getPuffinFilesCache()->maxSizeInBytes()), ChangeableWithoutRestart::Yes}}); /// `keeper_hosts` is not a regular config setting; it is derived from the `` config and follows /// it on config reload, so the live value diverges from the empty default stored in `ServerSettings`. diff --git a/src/Core/Settings.cpp b/src/Core/Settings.cpp index 7eb4d9f8055c..f29c787906a3 100644 --- a/src/Core/Settings.cpp +++ b/src/Core/Settings.cpp @@ -5593,6 +5593,14 @@ Minimum time of delay between 2 background compaction operations. )", 0) \ DECLARE(Seconds, iceberg_compaction_data_cleanup, 60 * 60 * 3, R"( The time after which the data will be deleted. +)", 0) \ + DECLARE(Bool, use_puffin_files_cache, true, R"( +If turned on, Iceberg reads may utilize the Puffin files cache for parsed puffin file content such as deletion vectors. + +Possible values: + +- 0 - Disabled +- 1 - Enabled )", 0) \ DECLARE(Bool, use_query_cache, false, R"( If turned on, `SELECT` queries may utilize the [query cache](../query-cache.md). Parameters [enable_reads_from_query_cache](#enable_reads_from_query_cache) diff --git a/src/Core/SettingsChangesHistory.cpp b/src/Core/SettingsChangesHistory.cpp index c0c5b6603ea5..c5ec63bc48d7 100644 --- a/src/Core/SettingsChangesHistory.cpp +++ b/src/Core/SettingsChangesHistory.cpp @@ -39,6 +39,11 @@ const VersionToSettingsChangesMap & getSettingsChangesHistory() /// controls new feature and it's 'true' by default, use 'false' as previous_value). /// It's used to implement `compatibility` setting (see https://github.com/ClickHouse/ClickHouse/issues/35972) /// Note: please check if the key already exists to prevent duplicate entries. + addSettingsChanges(settings_changes_history, "26.6.2.20001.altinityantalya", + { + {"use_puffin_files_cache", false, true, "Enables cache of parsed Puffin file content such as deletion vectors."}, + }); + addSettingsChanges(settings_changes_history, "26.6", { {"analyzer_compatibility_apply_final_to_all_joined_tables", true, false, "Fixed a bug in the analyzer where FINAL on the left-most table of a JOIN was incorrectly applied to the other joined tables as well. previous_value=true so `compatibility` with versions before 26.6 restores the old behavior."}, diff --git a/src/Formats/FormatFilterInfo.cpp b/src/Formats/FormatFilterInfo.cpp index b3e16b336098..2698990ea79a 100644 --- a/src/Formats/FormatFilterInfo.cpp +++ b/src/Formats/FormatFilterInfo.cpp @@ -82,7 +82,9 @@ FormatFilterInfo::FormatFilterInfo() = default; bool FormatFilterInfo::hasFilter() const { - return filter_actions_dag != nullptr; + /// Any of these can reduce the number of rows emitted by the reader pipeline. + /// Count-from-files cache must not be populated when they are present. + return filter_actions_dag != nullptr || row_level_filter != nullptr || prewhere_info != nullptr; } Block FormatFilterInfo::buildKeyConditionInputs( diff --git a/src/Formats/FormatFilterInfo.h b/src/Formats/FormatFilterInfo.h index f6ad9ac878b1..037058cc61b6 100644 --- a/src/Formats/FormatFilterInfo.h +++ b/src/Formats/FormatFilterInfo.h @@ -79,6 +79,7 @@ struct FormatFilterInfo std::exception_ptr init_exception; public: + /// True if WHERE / PREWHERE / row-policy filters may change the emitted row count. bool hasFilter() const; /// Creates `key_condition` and `additional_columns` with std::call_once semantics. diff --git a/src/Interpreters/ClusterFunctionReadTask.cpp b/src/Interpreters/ClusterFunctionReadTask.cpp index bd653629b504..de3ea864f445 100644 --- a/src/Interpreters/ClusterFunctionReadTask.cpp +++ b/src/Interpreters/ClusterFunctionReadTask.cpp @@ -92,6 +92,59 @@ void ClusterFunctionReadTaskResponse::serialize(WriteBuffer & out, size_t worker { auto protocol_version = std::min(static_cast(worker_protocol_version), static_cast(DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION)); + + /// Fail closed: protocol < 2 omits `schema_transform`, so workers would skip data-lake schema + /// evolution and return wrong columns / values. + if (protocol_version < DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_DATA_LAKE_METADATA + && data_lake_metadata.schema_transform + && !data_lake_metadata.schema_transform->getInputs().empty()) + { + throw Exception( + ErrorCodes::UNKNOWN_PROTOCOL, + "Worker protocol version {} cannot carry `schema_transform`, which is required for " + "distributed data-lake reads with schema evolution (minimum protocol version: {})", + protocol_version, + DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_DATA_LAKE_METADATA); + } + + /// Fail closed: downgrading would omit deletion / selection vectors and return deleted rows. + if (protocol_version < DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_EXCLUDED_ROWS + && hasNonEmptyExcludedRows(data_lake_metadata)) + { + throw Exception( + ErrorCodes::UNKNOWN_PROTOCOL, + "Worker protocol version {} cannot carry `excluded_rows`, which is required for distributed " + "reads with deletion vectors / selection vectors (minimum protocol version: {})", + protocol_version, + DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_EXCLUDED_ROWS); + } + + /// Fail closed: protocol < 3 omits `iceberg_info`, so workers rebuild a plain `ObjectInfo` + /// and lose Iceberg schema IDs / file format / delete transforms. + if (protocol_version < DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_ICEBERG_METADATA + && iceberg_info.has_value()) + { + throw Exception( + ErrorCodes::UNKNOWN_PROTOCOL, + "Worker protocol version {} cannot carry `iceberg_info` " + "(minimum protocol version: {})", + protocol_version, + DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_ICEBERG_METADATA); + } + + /// Fail closed: protocol < 4 omits `file_bucket_info`, so each bucket task becomes a full-file + /// read and bucket-split cluster queries return duplicated rows. + if (protocol_version < DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_FILE_BUCKETS_INFO + && file_bucket_info) + { + throw Exception( + ErrorCodes::UNKNOWN_PROTOCOL, + "Worker protocol version {} cannot carry `file_bucket_info`, which is required for " + "distributed bucket-split reads (minimum protocol version: {})", + protocol_version, + DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_FILE_BUCKETS_INFO); + } + writeVarUInt(protocol_version, out); writeStringBinary(path, out); diff --git a/src/Interpreters/Context.cpp b/src/Interpreters/Context.cpp index 6bc822d028b9..28152948079e 100644 --- a/src/Interpreters/Context.cpp +++ b/src/Interpreters/Context.cpp @@ -56,6 +56,7 @@ #include #include #include +#include #include #include #include @@ -592,6 +593,7 @@ struct ContextSharedPart : boost::noncopyable #if USE_PARQUET mutable ParquetMetadataCachePtr parquet_metadata_cache TSA_GUARDED_BY(mutex); /// Cache of deserialized parquet metadata files. #endif + mutable PuffinFilesCachePtr puffin_files_cache TSA_GUARDED_BY(mutex); /// Cache of parsed puffin file content. AsynchronousMetrics * asynchronous_metrics TSA_GUARDED_BY(mutex) = nullptr; /// Points to asynchronous metrics mutable PageCachePtr page_cache TSA_GUARDED_BY(mutex); /// Userspace page cache. ProcessList process_list; /// Executing queries at the moment. @@ -4783,6 +4785,52 @@ void Context::clearParquetMetadataCache() const } #endif +void Context::setPuffinFilesCache(const String & cache_policy, size_t max_size_in_bytes, size_t max_entries, double size_ratio) +{ + std::lock_guard lock(shared->mutex); + + if (shared->puffin_files_cache) + throw Exception(ErrorCodes::LOGICAL_ERROR, "Puffin files cache has been already created."); + + shared->puffin_files_cache = std::make_shared(cache_policy, max_size_in_bytes, max_entries, size_ratio); +} + +void Context::updatePuffinFilesCacheConfiguration(const Poco::Util::AbstractConfiguration & config, size_t max_cache_size) +{ + std::lock_guard lock(shared->mutex); + + if (!shared->puffin_files_cache) + throw Exception(ErrorCodes::LOGICAL_ERROR, "Puffin files cache was not created yet."); + + size_t size = config.getUInt64("puffin_files_cache_size", DEFAULT_PUFFIN_FILES_CACHE_MAX_SIZE); + size_t max_entries = config.getUInt64("puffin_files_cache_max_entries", DEFAULT_PUFFIN_FILES_CACHE_MAX_ENTRIES); + if (size > max_cache_size) + { + size = max_cache_size; + LOG_DEBUG(shared->log, "Lowered Puffin files cache size to {} because the system has limited RAM", formatReadableSizeWithBinarySuffix(size)); + } + shared->puffin_files_cache->setMaxSizeInBytes(size); + shared->puffin_files_cache->setMaxCount(max_entries); +} + +std::shared_ptr Context::getPuffinFilesCache() const +{ + SharedLockGuard lock(shared->mutex); + + if (!shared->puffin_files_cache) + throw Exception(ErrorCodes::LOGICAL_ERROR, "Puffin files cache was not created yet."); + return shared->puffin_files_cache; +} + +void Context::clearPuffinFilesCache() const +{ + auto cache = getPuffinFilesCache(); + + /// Clear the cache without holding context mutex to avoid blocking context for a long time + if (cache) + cache->clear(); +} + void Context::setQueryConditionCache(const String & cache_policy, size_t max_size_in_bytes, double size_ratio) { std::lock_guard lock(shared->mutex); diff --git a/src/Interpreters/Context.h b/src/Interpreters/Context.h index 413ef11f73da..805dedb707c4 100644 --- a/src/Interpreters/Context.h +++ b/src/Interpreters/Context.h @@ -110,6 +110,7 @@ class MMappedFileCache; class UncompressedCache; class IcebergMetadataFilesCache; class ParquetMetadataCache; +class PuffinFilesCache; class VectorSimilarityIndexCache; class TextIndexTokensCache; class TextIndexHeaderCache; @@ -1514,6 +1515,11 @@ class Context: public ContextData, public std::enable_shared_from_this void clearParquetMetadataCache() const; #endif + void setPuffinFilesCache(const String & cache_policy, size_t max_size_in_bytes, size_t max_entries, double size_ratio); + void updatePuffinFilesCacheConfiguration(const Poco::Util::AbstractConfiguration & config, size_t max_cache_size); + std::shared_ptr getPuffinFilesCache() const; + void clearPuffinFilesCache() const; + void setAllowedDisksForTableEngines(std::unordered_set && allowed_disks_) { allowed_disks = std::move(allowed_disks_); } const std::unordered_set & getAllowedDisksForTableEngines() const { return allowed_disks; } diff --git a/src/Interpreters/InterpreterSystemQuery.cpp b/src/Interpreters/InterpreterSystemQuery.cpp index 8bf2942e0d73..089e98ce3851 100644 --- a/src/Interpreters/InterpreterSystemQuery.cpp +++ b/src/Interpreters/InterpreterSystemQuery.cpp @@ -482,6 +482,10 @@ BlockIO InterpreterSystemQuery::execute() #else throw Exception(ErrorCodes::SUPPORT_IS_DISABLED, "The server was compiled without the support for Parquet"); #endif + case Type::CLEAR_PUFFIN_FILES_CACHE: + getContext()->checkAccess(AccessType::SYSTEM_DROP_PUFFIN_FILES_CACHE); + system_context->clearPuffinFilesCache(); + break; case Type::CLEAR_PRIMARY_INDEX_CACHE: getContext()->checkAccess(AccessType::SYSTEM_DROP_PRIMARY_INDEX_CACHE); system_context->clearPrimaryIndexCache(); @@ -2495,6 +2499,7 @@ AccessRightsElements InterpreterSystemQuery::getRequiredAccessForDDLOnCluster() case Type::CLEAR_ICEBERG_METADATA_CACHE: case Type::CLEAR_AVRO_SCHEMA_CACHE: case Type::CLEAR_PARQUET_METADATA_CACHE: + case Type::CLEAR_PUFFIN_FILES_CACHE: case Type::CLEAR_PRIMARY_INDEX_CACHE: case Type::CLEAR_MMAP_CACHE: case Type::CLEAR_QUERY_CONDITION_CACHE: diff --git a/src/Interpreters/tests/gtest_cluster_function_read_task.cpp b/src/Interpreters/tests/gtest_cluster_function_read_task.cpp new file mode 100644 index 000000000000..b471b53ed4b6 --- /dev/null +++ b/src/Interpreters/tests/gtest_cluster_function_read_task.cpp @@ -0,0 +1,314 @@ +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#if USE_PARQUET +#include +#include +#endif + +using namespace DB; + +namespace DB::ErrorCodes +{ +extern const int UNKNOWN_PROTOCOL; +} + +/// Cluster-protocol fail-closed behavior for Iceberg deletion vectors / equality / position +/// deletes and file buckets. Master-only APIs (`read_source_index`, +/// `derive_file_name_from_url_path`, `getIdentifier(bool)`) are intentionally not covered here. + +TEST(ClusterFunctionReadTaskResponse, RejectsSchemaTransformOnProtocolBeforeDataLakeMetadata) +{ + ClusterFunctionReadTaskResponse response; + response.path = "/path/file.parquet"; + response.data_lake_metadata.schema_transform + = std::make_shared(NamesAndTypesList{{"x", std::make_shared()}}); + + String serialized; + WriteBufferFromString out(serialized); + try + { + response.serialize(out, DBMS_CLUSTER_INITIAL_PROCESSING_PROTOCOL_VERSION); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::UNKNOWN_PROTOCOL); + EXPECT_NE(e.message().find("schema_transform"), std::string::npos); + } +} + +TEST(ClusterFunctionReadTaskResponse, AllowsEmptySchemaTransformOnProtocolBeforeDataLakeMetadata) +{ + ClusterFunctionReadTaskResponse response; + response.path = "/path/file.parquet"; + response.data_lake_metadata.schema_transform = std::make_shared(); + + String serialized; + WriteBufferFromString out(serialized); + response.serialize(out, DBMS_CLUSTER_INITIAL_PROCESSING_PROTOCOL_VERSION); + out.finalize(); + EXPECT_FALSE(serialized.empty()); +} + +TEST(ClusterFunctionReadTaskResponse, RoundTripsSchemaTransformOnSupportedProtocol) +{ + ClusterFunctionReadTaskResponse response; + response.path = "/path/file.parquet"; + response.data_lake_metadata.schema_transform + = std::make_shared(NamesAndTypesList{{"x", std::make_shared()}}); + + String serialized; + WriteBufferFromString out(serialized); + response.serialize(out, DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_DATA_LAKE_METADATA); + out.finalize(); + + ReadBufferFromString in(serialized); + ClusterFunctionReadTaskResponse deserialized; + deserialized.deserialize(in); + + ASSERT_TRUE(deserialized.data_lake_metadata.schema_transform); + EXPECT_FALSE(deserialized.data_lake_metadata.schema_transform->getInputs().empty()); +} + +TEST(ClusterFunctionReadTaskResponse, RejectsNonEmptyExcludedRowsOnOldProtocol) +{ + ClusterFunctionReadTaskResponse response; + response.path = "/path/file.parquet"; + response.data_lake_metadata.excluded_rows = std::make_shared(); + response.data_lake_metadata.excluded_rows->add(7); + + String serialized; + WriteBufferFromString out(serialized); + try + { + response.serialize(out, DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_FILE_BUCKETS_INFO); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::UNKNOWN_PROTOCOL); + EXPECT_NE(e.message().find("excluded_rows"), std::string::npos); + } +} + +TEST(ClusterFunctionReadTaskResponse, AllowsEmptyExcludedRowsOnOldProtocol) +{ + ClusterFunctionReadTaskResponse response; + response.path = "/path/file.parquet"; + response.data_lake_metadata.excluded_rows = std::make_shared(); + + String serialized; + WriteBufferFromString out(serialized); + response.serialize(out, DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_FILE_BUCKETS_INFO); + out.finalize(); + EXPECT_FALSE(serialized.empty()); +} + +TEST(ClusterFunctionReadTaskResponse, RoundTripsExcludedRowsOnSupportedProtocol) +{ + ClusterFunctionReadTaskResponse response; + response.path = "/path/file.parquet"; + response.data_lake_metadata.excluded_rows = std::make_shared(); + response.data_lake_metadata.excluded_rows->add(3); + response.data_lake_metadata.excluded_rows->add(9); + + String serialized; + WriteBufferFromString out(serialized); + response.serialize(out, DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_EXCLUDED_ROWS); + out.finalize(); + + ReadBufferFromString in(serialized); + ClusterFunctionReadTaskResponse deserialized; + deserialized.deserialize(in); + + ASSERT_TRUE(deserialized.data_lake_metadata.excluded_rows); + EXPECT_EQ(deserialized.data_lake_metadata.excluded_rows->size(), 2u); + EXPECT_TRUE(deserialized.data_lake_metadata.excluded_rows->rb_contains(3)); + EXPECT_TRUE(deserialized.data_lake_metadata.excluded_rows->rb_contains(9)); +} + +TEST(ClusterFunctionReadTaskResponse, RejectsEqualityDeletesOnProtocolBeforeIcebergMetadata) +{ + ClusterFunctionReadTaskResponse response; + response.path = "/path/file.parquet"; + response.iceberg_info = Iceberg::IcebergObjectSerializableInfo{}; + response.iceberg_info->equality_deletes_objects.push_back( + Iceberg::EqualityDeleteObject{ + .file_path = "/path/eq.parquet", + .file_format = "PARQUET", + .equality_ids = std::vector{1}, + .schema_id = 0, + }); + + String serialized; + WriteBufferFromString out(serialized); + try + { + response.serialize(out, DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_DATA_LAKE_METADATA); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::UNKNOWN_PROTOCOL); + EXPECT_NE(e.message().find("iceberg_info"), std::string::npos); + } +} + +TEST(ClusterFunctionReadTaskResponse, RejectsPositionDeletesOnProtocolBeforeIcebergMetadata) +{ + ClusterFunctionReadTaskResponse response; + response.path = "/path/file.parquet"; + response.iceberg_info = Iceberg::IcebergObjectSerializableInfo{}; + response.iceberg_info->position_deletes_objects.push_back( + Iceberg::PositionDeleteObject{ + .file_path = "/path/pos.parquet", + .file_format = "PARQUET", + .reference_data_file_path = std::nullopt, + .sequence_number = 1, + }); + + String serialized; + WriteBufferFromString out(serialized); + try + { + response.serialize(out, DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_DATA_LAKE_METADATA); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::UNKNOWN_PROTOCOL); + EXPECT_NE(e.message().find("iceberg_info"), std::string::npos); + } +} + +TEST(ClusterFunctionReadTaskResponse, RejectsIcebergInfoWithoutDeletesOnProtocolBeforeIcebergMetadata) +{ + ClusterFunctionReadTaskResponse response; + response.path = "/path/file.parquet"; + response.iceberg_info = Iceberg::IcebergObjectSerializableInfo{}; + response.iceberg_info->underlying_format_read_schema_id = 1; + response.iceberg_info->schema_id_relevant_to_iterator = 2; + response.iceberg_info->file_format = "PARQUET"; + + String serialized; + WriteBufferFromString out(serialized); + try + { + response.serialize(out, DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_DATA_LAKE_METADATA); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::UNKNOWN_PROTOCOL); + EXPECT_NE(e.message().find("iceberg_info"), std::string::npos); + } +} + +TEST(ClusterFunctionReadTaskResponse, RoundTripsIcebergDeletesOnSupportedProtocol) +{ + ClusterFunctionReadTaskResponse response; + response.path = "/path/file.parquet"; + response.iceberg_info = Iceberg::IcebergObjectSerializableInfo{}; + response.iceberg_info->data_object_file_path_key + = Iceberg::IcebergPathFromMetadata::deserialize("s3://bucket/path/file.parquet"); + response.iceberg_info->file_format = "PARQUET"; + response.iceberg_info->equality_deletes_objects.push_back( + Iceberg::EqualityDeleteObject{ + .file_path = "/path/eq.parquet", + .file_format = "PARQUET", + .equality_ids = std::vector{1, 2}, + .schema_id = 7, + }); + response.iceberg_info->position_deletes_objects.push_back( + Iceberg::PositionDeleteObject{ + .file_path = "/path/pos.parquet", + .file_format = "PARQUET", + .reference_data_file_path = std::nullopt, + .sequence_number = 42, + }); + + String serialized; + WriteBufferFromString out(serialized); + response.serialize(out, DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_ICEBERG_METADATA); + out.finalize(); + + ReadBufferFromString in(serialized); + ClusterFunctionReadTaskResponse deserialized; + deserialized.deserialize(in); + + ASSERT_TRUE(deserialized.iceberg_info.has_value()); + ASSERT_EQ(deserialized.iceberg_info->equality_deletes_objects.size(), 1u); + EXPECT_EQ(deserialized.iceberg_info->equality_deletes_objects[0].file_path, "/path/eq.parquet"); + ASSERT_EQ(deserialized.iceberg_info->position_deletes_objects.size(), 1u); + EXPECT_EQ(deserialized.iceberg_info->position_deletes_objects[0].file_path, "/path/pos.parquet"); +} + +#if USE_PARQUET + +TEST(ClusterFunctionReadTaskResponse, RejectsFileBucketInfoOnProtocolBeforeFileBuckets) +{ + ClusterFunctionReadTaskResponse response; + response.path = "/path/file.parquet"; + response.file_bucket_info = std::make_shared(std::vector{0, 1}); + + String serialized; + WriteBufferFromString out(serialized); + try + { + response.serialize(out, DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_ICEBERG_METADATA); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::UNKNOWN_PROTOCOL); + EXPECT_NE(e.message().find("file_bucket_info"), std::string::npos); + } +} + +TEST(ClusterFunctionReadTaskResponse, AllowsMissingFileBucketInfoOnProtocolBeforeFileBuckets) +{ + ClusterFunctionReadTaskResponse response; + response.path = "/path/file.parquet"; + + String serialized; + WriteBufferFromString out(serialized); + response.serialize(out, DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_ICEBERG_METADATA); + out.finalize(); + EXPECT_FALSE(serialized.empty()); +} + +TEST(ClusterFunctionReadTaskResponse, RoundTripsFileBucketInfoOnSupportedProtocol) +{ + tryRegisterFormats(); + + ClusterFunctionReadTaskResponse response; + response.path = "/path/file.parquet"; + response.file_bucket_info = std::make_shared(std::vector{2, 5}); + + String serialized; + WriteBufferFromString out(serialized); + response.serialize(out, DBMS_CLUSTER_PROCESSING_PROTOCOL_VERSION_WITH_FILE_BUCKETS_INFO); + out.finalize(); + + ReadBufferFromString in(serialized); + ClusterFunctionReadTaskResponse deserialized; + deserialized.deserialize(in); + + ASSERT_TRUE(deserialized.file_bucket_info); + auto * parquet_buckets = dynamic_cast(deserialized.file_bucket_info.get()); + ASSERT_TRUE(parquet_buckets); + EXPECT_EQ(parquet_buckets->row_group_ids, (std::vector{2, 5})); +} + +#endif diff --git a/src/Parsers/ASTSystemQuery.cpp b/src/Parsers/ASTSystemQuery.cpp index 773ece4517d9..01740135c3f9 100644 --- a/src/Parsers/ASTSystemQuery.cpp +++ b/src/Parsers/ASTSystemQuery.cpp @@ -604,6 +604,7 @@ void ASTSystemQuery::formatImpl(WriteBuffer & ostr, const FormatSettings & setti case Type::CLEAR_ICEBERG_METADATA_CACHE: case Type::CLEAR_PARQUET_METADATA_CACHE: case Type::CLEAR_AVRO_SCHEMA_CACHE: + case Type::CLEAR_PUFFIN_FILES_CACHE: case Type::DROP_OBJECT_STORAGE_LIST_OBJECTS_CACHE: case Type::RESET_COVERAGE: case Type::RESTART_REPLICAS: diff --git a/src/Parsers/ASTSystemQuery.h b/src/Parsers/ASTSystemQuery.h index 517b1c6304c9..f83e21689c5c 100644 --- a/src/Parsers/ASTSystemQuery.h +++ b/src/Parsers/ASTSystemQuery.h @@ -45,6 +45,7 @@ class ASTSystemQuery : public IAST, public ASTQueryWithOnCluster CLEAR_COMPILED_EXPRESSION_CACHE, CLEAR_ICEBERG_METADATA_CACHE, CLEAR_PARQUET_METADATA_CACHE, + CLEAR_PUFFIN_FILES_CACHE, CLEAR_FILESYSTEM_CACHE, CLEAR_DISTRIBUTED_CACHE, CLEAR_DISK_METADATA_CACHE, diff --git a/src/Parsers/ParserSystemQuery.cpp b/src/Parsers/ParserSystemQuery.cpp index 07b79465ea04..b2ed7392ee6c 100644 --- a/src/Parsers/ParserSystemQuery.cpp +++ b/src/Parsers/ParserSystemQuery.cpp @@ -285,6 +285,8 @@ bool ParserSystemQuery::parseImpl(IParser::Pos & pos, ASTPtr & node, Expected & {"DROP COMPILED EXPRESSION CACHE", Type::CLEAR_COMPILED_EXPRESSION_CACHE}, {"DROP ICEBERG METADATA CACHE", Type::CLEAR_ICEBERG_METADATA_CACHE}, {"DROP PARQUET METADATA CACHE", Type::CLEAR_PARQUET_METADATA_CACHE}, + {"DROP PUFFIN FILES CACHE", Type::CLEAR_PUFFIN_FILES_CACHE}, + {"DROP PUFFIN_FILES_CACHE", Type::CLEAR_PUFFIN_FILES_CACHE}, {"DROP FILESYSTEM CACHE", Type::CLEAR_FILESYSTEM_CACHE}, {"DROP DISTRIBUTED CACHE", Type::CLEAR_DISTRIBUTED_CACHE}, {"DROP DISK METADATA CACHE", Type::CLEAR_DISK_METADATA_CACHE}, diff --git a/src/Processors/Formats/Impl/Parquet/Reader.cpp b/src/Processors/Formats/Impl/Parquet/Reader.cpp index e0422d65ce2d..4fead45ac047 100644 --- a/src/Processors/Formats/Impl/Parquet/Reader.cpp +++ b/src/Processors/Formats/Impl/Parquet/Reader.cpp @@ -8,6 +8,7 @@ #include #include #include +#include #include #include #include @@ -321,6 +322,42 @@ void Reader::getHyperrectangleForRowGroup(const parq::RowGroup * meta, Hyperrect } } +std::vector buildRowGroupGlobalOffsets(const parq::FileMetaData & file_metadata) +{ + if (file_metadata.num_rows < 0) + throw Exception(ErrorCodes::INCORRECT_DATA, "Parquet file has negative row count: {}", file_metadata.num_rows); + + const size_t num_row_groups = file_metadata.row_groups.size(); + std::vector global_offsets(num_row_groups + 1, 0); + UInt64 total_rows = 0; + + for (size_t i = 0; i < num_row_groups; ++i) + { + const Int64 num_rows = file_metadata.row_groups[i].num_rows; + if (num_rows < 0) + throw Exception(ErrorCodes::INCORRECT_DATA, "Parquet row group {} has negative row count: {}", i, num_rows); + + UInt64 next_total = 0; + if (common::addOverflow(total_rows, static_cast(num_rows), next_total)) + { + throw Exception( + ErrorCodes::INCORRECT_DATA, + "Parquet row group row counts overflow when computing global offsets (at row group {})", + i); + } + + total_rows = next_total; + global_offsets[i + 1] = static_cast(total_rows); + } + + /// Do not require the row-group sum to equal `FileMetaData.num_rows`. Some writers leave a + /// stale or inconsistent file-level count; global offsets and deletion-vector positions are + /// defined by the row-group layout. This helper runs on every ParquetV3 read, so rejecting + /// mismatches would break previously readable files. + + return global_offsets; +} + void Reader::prefilterAndInitRowGroups(const std::optional> & row_groups_to_read) { extended_sample_block = *sample_block; @@ -376,19 +413,15 @@ void Reader::prefilterAndInitRowGroups(const std::optional global_offsets = buildRowGroupGlobalOffsets(file_metadata); for (size_t row_group_idx = 0; row_group_idx < file_metadata.row_groups.size(); ++row_group_idx) { const auto * meta = &file_metadata.row_groups[row_group_idx]; - if (meta->num_rows < 0) - throw Exception(ErrorCodes::INCORRECT_DATA, "Row group {} has negative row count: {}", row_group_idx, meta->num_rows); if (meta->num_rows == 0) continue; /// Empty row groups are valid in Parquet; skip them. if (meta->columns.size() != total_primitive_columns_in_file) throw Exception(ErrorCodes::INCORRECT_DATA, "Row group {} has unexpected number of columns: {} != {}", row_group_idx, meta->columns.size(), total_primitive_columns_in_file); - total_rows += size_t(meta->num_rows); // before potentially skipping the row group - Hyperrectangle hyperrectangle(extended_sample_block.columns(), Range::createWholeUniverse()); if (options.format.parquet.filter_push_down && format_filter_info->key_condition) { @@ -402,7 +435,7 @@ void Reader::prefilterAndInitRowGroups(const std::optionalcontains(row_group_idx); row_group.row_group_idx = row_group_idx; - row_group.start_global_row_idx = total_rows - size_t(meta->num_rows); + row_group.start_global_row_idx = global_offsets[row_group_idx]; row_group.columns.resize(primitive_columns.size()); row_group.hyperrectangle = std::move(hyperrectangle); diff --git a/src/Processors/Formats/Impl/Parquet/Reader.h b/src/Processors/Formats/Impl/Parquet/Reader.h index 0ac46ac11f31..36841514dc0f 100644 --- a/src/Processors/Formats/Impl/Parquet/Reader.h +++ b/src/Processors/Formats/Impl/Parquet/Reader.h @@ -571,4 +571,9 @@ struct Reader void readRowsInPage(size_t end_row_idx, ColumnSubchunk & subchunk, ColumnChunk & column, const PrimitiveColumnInfo & column_info, const RowSubgroup * row_subgroup = nullptr); }; +/// Prefix offsets of row groups in the file; result size is `row_groups.size() + 1`. +/// Throws `INCORRECT_DATA` on negative counts or size overflow. A mismatch between the +/// row-group sum and `FileMetaData.num_rows` is tolerated: offsets follow the row-group layout. +std::vector buildRowGroupGlobalOffsets(const parq::FileMetaData & file_metadata); + } diff --git a/src/Processors/Formats/Impl/ParquetV3BlockInputFormat.cpp b/src/Processors/Formats/Impl/ParquetV3BlockInputFormat.cpp index edbf421ccbeb..6a238834caec 100644 --- a/src/Processors/Formats/Impl/ParquetV3BlockInputFormat.cpp +++ b/src/Processors/Formats/Impl/ParquetV3BlockInputFormat.cpp @@ -25,6 +25,7 @@ namespace DB namespace ErrorCodes { extern const int LOGICAL_ERROR; + extern const int INCORRECT_DATA; } static Parquet::ReadOptions convertReadOptions(const FormatSettings & format_settings) @@ -120,23 +121,65 @@ parquet::format::FileMetaData ParquetV3BlockInputFormat::getFileMetadata(Parquet } } -Chunk ParquetV3BlockInputFormat::read() +void ParquetV3BlockInputFormat::prepareNeedOnlyCountRowGroups(const parquet::format::FileMetaData & file_metadata) { - if (need_only_count) + need_only_count_row_groups.clear(); + need_only_count_next = 0; + + if (!buckets_to_read) { - if (reported_count) - return {}; + /// Unbucketed need_only_count historically returns one chunk from `FileMetaData.num_rows`. + /// Keep that for plain Parquet COUNT (and Iceberg without row-group buckets). Per-row-group + /// spans below are only for bucketed reads, where each task must count its assigned groups + /// instead of the whole file. + if (file_metadata.num_rows < 0) + throw Exception(ErrorCodes::INCORRECT_DATA, "Parquet file has negative row count: {}", file_metadata.num_rows); + + need_only_count_row_groups.push_back( + {.row_num_offset = 0, .num_rows = static_cast(file_metadata.num_rows)}); + return; + } - /// Don't init Reader and ReadManager if we only need file metadata. - Parquet::Prefetcher temp_prefetcher; - temp_prefetcher.init(in, read_options, parser_shared_resources); - parquet::format::FileMetaData file_metadata = getFileMetadata(temp_prefetcher); + const std::vector global_offsets = Parquet::buildRowGroupGlobalOffsets(file_metadata); + const size_t num_row_groups = file_metadata.row_groups.size(); + for (size_t row_group_id : buckets_to_read->row_group_ids) + { + if (row_group_id >= num_row_groups) + throw Exception( + ErrorCodes::INCORRECT_DATA, + "Parquet bucket row group {} is out of range (file has {} row groups)", + row_group_id, + num_row_groups); + + const size_t num_rows = static_cast(file_metadata.row_groups[row_group_id].num_rows); + if (num_rows == 0) + continue; - auto chunk = getChunkForCount(size_t(file_metadata.num_rows)); - chunk.getChunkInfos().add(std::make_shared(0)); + need_only_count_row_groups.push_back( + {.row_num_offset = global_offsets[row_group_id], .num_rows = num_rows}); + } +} + +Chunk ParquetV3BlockInputFormat::read() +{ + if (need_only_count) + { + if (!need_only_count_prepared) + { + /// Don't init Reader and ReadManager if we only need file metadata. + Parquet::Prefetcher temp_prefetcher; + temp_prefetcher.init(in, read_options, parser_shared_resources); + prepareNeedOnlyCountRowGroups(getFileMetadata(temp_prefetcher)); + need_only_count_prepared = true; + } + + if (need_only_count_next >= need_only_count_row_groups.size()) + return {}; - reported_count = true; + const auto & row_group = need_only_count_row_groups[need_only_count_next++]; + auto chunk = getChunkForCount(row_group.num_rows); + chunk.getChunkInfos().add(std::make_shared(row_group.row_num_offset)); return chunk; } @@ -199,6 +242,9 @@ void ParquetV3BlockInputFormat::resetParser() reader.reset(); } previous_block_missing_values.clear(); + need_only_count_prepared = false; + need_only_count_next = 0; + need_only_count_row_groups.clear(); IInputFormat::resetParser(); } diff --git a/src/Processors/Formats/Impl/ParquetV3BlockInputFormat.h b/src/Processors/Formats/Impl/ParquetV3BlockInputFormat.h index 87a5192ae641..2c5baf00ea92 100644 --- a/src/Processors/Formats/Impl/ParquetV3BlockInputFormat.h +++ b/src/Processors/Formats/Impl/ParquetV3BlockInputFormat.h @@ -81,12 +81,21 @@ class ParquetV3BlockInputFormat final : public IInputFormat std::mutex reader_mutex; std::optional reader; - bool reported_count = false; // if need_only_count + bool need_only_count_prepared = false; + size_t need_only_count_next = 0; + + struct NeedOnlyCountRowGroup + { + size_t row_num_offset = 0; + size_t num_rows = 0; + }; + std::vector need_only_count_row_groups; BlockMissingValues previous_block_missing_values; size_t previous_approx_bytes_read_for_chunk = 0; void initializeIfNeeded(); + void prepareNeedOnlyCountRowGroups(const parquet::format::FileMetaData & file_metadata); std::shared_ptr buckets_to_read; parquet::format::FileMetaData getFileMetadata(Parquet::Prefetcher & prefetcher) const; diff --git a/src/Processors/Formats/Impl/PuffinBlockInputFormat.cpp b/src/Processors/Formats/Impl/PuffinBlockInputFormat.cpp index e742a400410d..db9559027330 100644 --- a/src/Processors/Formats/Impl/PuffinBlockInputFormat.cpp +++ b/src/Processors/Formats/Impl/PuffinBlockInputFormat.cpp @@ -24,7 +24,9 @@ #include #include #include +#include #include +#include #include #include #include @@ -32,9 +34,16 @@ #include #include #include +#include #include +namespace ProfileEvents +{ +extern const Event PuffinFilesRead; +extern const Event PuffinFileReadMicroseconds; +} + namespace DB { @@ -44,26 +53,33 @@ namespace ErrorCodes extern const int LZ4_DECODER_FAILED; } +constexpr Int64 DELETION_VECTOR_MAX_POSITION = 0x7FFFFFFE80000000LL; +constexpr Int32 DELETION_VECTOR_MAX_KEY = std::numeric_limits::max() - 1; + +static UInt64 positionFromKeyAndSubPosition(UInt32 key, UInt32 sub_position) +{ + return (static_cast(key) << 32) | static_cast(sub_position); +} + namespace { +struct ScopedPuffinFileReadProfileEvent +{ + ProfileEventTimeIncrement watch; + + ScopedPuffinFileReadProfileEvent() + : watch(ProfileEvents::PuffinFileReadMicroseconds) + { + ProfileEvents::increment(ProfileEvents::PuffinFilesRead); + } +}; + constexpr UInt8 PUFFIN_MAGIC[4] = {0x50, 0x46, 0x41, 0x31}; constexpr UInt8 PUFFIN_FOOTER_COMPRESSED_FLAG = 0x01; -constexpr size_t PUFFIN_FOOTER_TRAILER_SIZE = 12; constexpr size_t PUFFIN_FOOTER_LZ4_MAX_RATIO = 255; -constexpr size_t PUFFIN_FOOTER_MAX_PAYLOAD_SIZE = 16 * 1024 * 1024; -constexpr UInt64 PUFFIN_DV_MAX_MATERIALIZED_POSITIONS = 100'000'000; -constexpr size_t PUFFIN_DV_MAX_BLOB_SIZE = 2ULL * 1024 * 1024 * 1024; -constexpr UInt8 DELETION_VECTOR_MAGIC[4] = {0xD1, 0xD3, 0x39, 0x64}; -constexpr Int64 DELETION_VECTOR_MAX_POSITION = 0x7FFFFFFE80000000LL; -constexpr Int32 DELETION_VECTOR_MAX_KEY = std::numeric_limits::max() - 1; constexpr const char * PUFFIN_DELETION_VECTOR_BLOB_TYPE = "deletion-vector-v1"; -UInt64 positionFromKeyAndSubPosition(UInt32 key, UInt32 sub_position) -{ - return (static_cast(key) << 32) | static_cast(sub_position); -} - void checkMagic(const UInt8 * p, const char * context) { if (std::memcmp(p, PUFFIN_MAGIC, 4) != 0) @@ -253,28 +269,6 @@ String requireBlobMetadataString(const Poco::JSON::Object::Ptr & blob_obj, const return requireJSONStringValue(blob_obj->get(field_name), blob_index, field_name); } -void requireDeletionVectorV1Properties(const PuffinBlob & blob, size_t blob_index) -{ - static constexpr const char * required_properties[] = {"referenced-data-file", "cardinality"}; - for (const char * key : required_properties) - { - auto it = blob.properties.find(key); - if (it == blob.properties.end() || it->second.empty()) - throw Exception( - ErrorCodes::BAD_ARGUMENTS, - "Puffin blob {}: deletion-vector-v1 missing required property '{}'", - blob_index, - key); - } - - UInt64 cardinality = 0; - if (!tryParse(cardinality, blob.properties.at("cardinality"))) - throw Exception( - ErrorCodes::BAD_ARGUMENTS, - "Puffin blob {}: deletion-vector-v1 property 'cardinality' must be an unsigned integer", - blob_index); -} - void parseStringValuedProperties( const Poco::JSON::Object::Ptr & props_obj, std::map * out, @@ -387,12 +381,13 @@ std::vector parseFooterJSON(const String & footer_json, size_t blob_ } else { - throw Exception( - ErrorCodes::BAD_ARGUMENTS, - "Puffin blob {}: unsupported blob type '{}', only '{}' is supported", - i, - blob.type, - PUFFIN_DELETION_VECTOR_BLOB_TYPE); + /// Puffin allows arbitrary blob types in one file (indexes, sketches, DVs, ...). + /// Keep common metadata so Iceberg can bind a DV by offset/length; do not require + /// deletion-vector properties for non-DV entries. + if (blob_obj->has("compression-codec") && !blob_obj->isNull("compression-codec")) + blob.compression_codec = requireBlobMetadataString(blob_obj, "compression-codec", i); + + parseBlobProperties(blob_obj, blob, i, /*required=*/false); } requireBlobMetadataField(blob_obj, "fields", i); @@ -410,8 +405,10 @@ std::vector parseFooterJSON(const String & footer_json, size_t blob_ return blobs; } -std::vector readPuffinFooterFromSeekable(SeekableReadBuffer & seekable, size_t file_size) +std::vector readPuffinFooterFromSeekableImpl(SeekableReadBuffer & seekable, size_t file_size) { + ScopedPuffinFileReadProfileEvent profile_event; + if (file_size < 16) throw Exception(ErrorCodes::BAD_ARGUMENTS, "Puffin file too small"); @@ -485,7 +482,7 @@ PuffinFooter readPuffinFooter(ReadBuffer & buf, bool seekable_read) if (seekable_read && seekable && seekable->checkIfActuallySeekable() && file_size_opt) { - result.blobs = readPuffinFooterFromSeekable(*seekable, *file_size_opt); + result.blobs = readPuffinFooterFromSeekableImpl(*seekable, *file_size_opt); } else { @@ -495,15 +492,10 @@ PuffinFooter readPuffinFooter(ReadBuffer & buf, bool seekable_read) throw Exception(ErrorCodes::BAD_ARGUMENTS, "Puffin file too small"); checkMagic(result.data.data(), "header"); - std::vector tmp(DEFAULT_BLOCK_SIZE); - while (!buf.eof()) - { - size_t n = buf.read(reinterpret_cast(tmp.data()), tmp.size()); - result.data.insert(result.data.end(), tmp.data(), tmp.data() + n); - } + appendReadBufferWithAbsoluteSizeLimit(buf, result.data, PUFFIN_NON_SEEKABLE_MAX_BUFFERED_SIZE); ReadBufferFromMemory mem_buf(result.data.data(), result.data.size()); - result.blobs = readPuffinFooterFromSeekable(mem_buf, result.data.size()); + result.blobs = readPuffinFooterFromSeekableImpl(mem_buf, result.data.size()); } return result; @@ -554,60 +546,159 @@ void readDeletionVectorEnvelopePrefix( } String readDeletionVectorBlobBytes( - const PuffinBlob & blob, ReadBuffer & buf, const std::vector & data, bool seekable_read) + const PuffinBlob & blob, + ReadBuffer & buf, + const std::vector & data, + bool seekable_read, + UInt64 expected_cardinality) { - if (blob.length < 0) - throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector blob length is negative"); - - if (static_cast(blob.length) > PUFFIN_DV_MAX_BLOB_SIZE) - throw Exception( - ErrorCodes::BAD_ARGUMENTS, - "Deletion vector blob length {} exceeds absolute limit {}", - blob.length, - PUFFIN_DV_MAX_BLOB_SIZE); + ScopedPuffinFileReadProfileEvent profile_event; - if (static_cast(blob.length) < 12) - throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector blob is too small"); + /// Fail closed before envelope peek / full allocate — shared with Iceberg + /// `readDeletionVectorFromPuffin`. `deserializeDeletionVectorV1` still re-checks. + checkDeletionVectorBlobReadLimits(blob.length, expected_cardinality); UInt8 header[8]; readDeletionVectorEnvelopePrefix(blob, buf, data, seekable_read, header); + validateDeletionVectorEnvelope(header, blob.length); - ReadBufferFromMemory header_buf(reinterpret_cast(header), sizeof(header)); - UInt32 combined_length = 0; - readBinaryBigEndian(combined_length, header_buf); - if (std::memcmp(header + sizeof(UInt32), DELETION_VECTOR_MAGIC, sizeof(DELETION_VECTOR_MAGIC)) != 0) - throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid deletion vector magic"); + return readPuffinBlobBytes(blob, buf, data, seekable_read); +} - if (combined_length < sizeof(DELETION_VECTOR_MAGIC)) - throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid deletion vector combined length: {}", combined_length); +NamesAndTypesList getPuffinMetadataSchema() +{ + return { + {"blob_type", std::make_shared()}, + {"snapshot_id", std::make_shared()}, + {"sequence_number", std::make_shared()}, + {"fields", std::make_shared(std::make_shared())}, + {"offset", std::make_shared()}, + {"length", std::make_shared()}, + {"compression_codec", std::make_shared()}, + {"properties", std::make_shared(std::make_shared(), std::make_shared())}, + }; +} - UInt64 expected_blob_size = 0; - if (common::addOverflow(static_cast(combined_length), UInt64{8}, expected_blob_size)) - throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid deletion vector combined length: {}", combined_length); +NamesAndTypesList getPuffinSchema() +{ + return { + {"referenced_data_file", std::make_shared()}, + {"deleted_rows", std::make_shared(std::make_shared())}, + }; +} + +void checkPuffinFormatHeader(const Block & header, const NamesAndTypesList & expected_schema, const char * format_name) +{ + std::unordered_map name_to_type; + for (const auto & [name, type] : expected_schema) + name_to_type[name] = type; + + String allowed_columns; + for (const auto & [name, type] : expected_schema) + { + if (!allowed_columns.empty()) + allowed_columns += ", "; + allowed_columns += name; + } + + for (const auto & [name, type] : header.getNamesAndTypes()) + { + auto it = name_to_type.find(name); + if (it == name_to_type.end()) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Unexpected column: {}. {} format allows only the next columns: {}", + name, + format_name, + allowed_columns); + + if (!it->second->equals(*type)) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Unexpected type {} for column {}. Expected type: {}", + type->getName(), + name, + it->second->getName()); + } +} + +void checkPuffinMetadataHeader(const Block & header) +{ + checkPuffinFormatHeader(header, getPuffinMetadataSchema(), "PuffinMetadata"); +} + +void checkPuffinHeader(const Block & header) +{ + checkPuffinFormatHeader(header, getPuffinSchema(), "Puffin"); +} - if (static_cast(blob.length) != expected_blob_size) +} + +UInt64 requireDeletionVectorV1Properties(const PuffinBlob & blob, size_t blob_index) +{ + /// Puffin v1: snapshot-id and sequence-number are unknown when the file is written and must be -1. + if (blob.snapshot_id != -1 || blob.sequence_number != -1) throw Exception( ErrorCodes::BAD_ARGUMENTS, - "Deletion vector blob size {} does not match combined length {}", - blob.length, - combined_length); + "Puffin blob {}: deletion-vector-v1 snapshot-id and sequence-number must be -1", + blob_index); - return readPuffinBlobBytes(blob, buf, data, seekable_read); + validateDeletionVectorV1Fields(blob.fields, blob_index); + + static constexpr const char * required_properties[] = {"referenced-data-file", "cardinality"}; + for (const char * key : required_properties) + { + auto it = blob.properties.find(key); + if (it == blob.properties.end() || it->second.empty()) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Puffin blob {}: deletion-vector-v1 missing required property '{}'", + blob_index, + key); + } + + UInt64 cardinality = 0; + if (!tryParse(cardinality, blob.properties.at("cardinality"))) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Puffin blob {}: deletion-vector-v1 property 'cardinality' must be an unsigned integer", + blob_index); + + return cardinality; } +namespace +{ + roaring::Roaring readRoaringPortableSafe(const char * data, size_t size, Int32 key) { + roaring::Roaring bitmap; try { - return roaring::Roaring::readSafe(data, size); + bitmap = roaring::Roaring::readSafe(data, size); } catch (const std::exception & e) { throw Exception(ErrorCodes::BAD_ARGUMENTS, "Failed to deserialize deletion vector roaring bitmap at key {}: {}", key, e.what()); } + + /// `readSafe` only bounds the read; CRoaring requires internal validation before use on untrusted input. + const char * reason = nullptr; + if (!roaring::api::roaring_bitmap_internal_validate(&bitmap.roaring, &reason)) + { + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Deletion vector roaring bitmap at key {} failed internal validation: {}", + key, + reason ? reason : "unknown"); + } + + return bitmap; +} + } -void deserializeRoaringPositionBitmap(std::string_view bytes, UInt64 expected_cardinality, ColumnUInt64 & positions) +std::vector deserializeRoaringPositionBitmap(std::string_view bytes, std::optional expected_cardinality) { if (bytes.size() < sizeof(Int64)) throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector bitmap is too small"); @@ -623,6 +714,10 @@ void deserializeRoaringPositionBitmap(std::string_view bytes, UInt64 expected_ca if (bitmap_count < 0 || bitmap_count > std::numeric_limits::max()) throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid deletion vector bitmap count: {}", bitmap_count); + std::vector positions; + if (expected_cardinality.has_value()) + positions.reserve(*expected_cardinality); + Int32 last_key = -1; Int32 remaining_count = static_cast(bitmap_count); UInt64 running_cardinality = 0; @@ -647,29 +742,32 @@ void deserializeRoaringPositionBitmap(std::string_view bytes, UInt64 expected_ca if (bitmap_size > remaining) throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector roaring bitmap at key {} exceeds blob size", key); - const UInt64 bitmap_cardinality = bitmap.cardinality(); - UInt64 new_running_cardinality = 0; - if (common::addOverflow(running_cardinality, bitmap_cardinality, new_running_cardinality)) - throw Exception( - ErrorCodes::BAD_ARGUMENTS, - "Deletion vector cardinality exceeds declared cardinality {}", - expected_cardinality); + if (expected_cardinality.has_value()) + { + const UInt64 bitmap_cardinality = bitmap.cardinality(); + UInt64 new_running_cardinality = 0; + if (common::addOverflow(running_cardinality, bitmap_cardinality, new_running_cardinality)) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Deletion vector cardinality exceeds declared cardinality {}", + *expected_cardinality); - if (new_running_cardinality > expected_cardinality) - throw Exception( - ErrorCodes::BAD_ARGUMENTS, - "Deletion vector cardinality {} exceeds declared cardinality {}", - new_running_cardinality, - expected_cardinality); + if (new_running_cardinality > *expected_cardinality) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Deletion vector cardinality {} exceeds declared cardinality {}", + new_running_cardinality, + *expected_cardinality); - running_cardinality = new_running_cardinality; + running_cardinality = new_running_cardinality; + } for (UInt32 sub_position : bitmap) { const UInt64 position = positionFromKeyAndSubPosition(static_cast(key), sub_position); if (position > static_cast(DELETION_VECTOR_MAX_POSITION)) throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector position {} is out of supported range", position); - positions.insertValue(position); + positions.push_back(position); } ptr += bitmap_size; @@ -681,12 +779,21 @@ void deserializeRoaringPositionBitmap(std::string_view bytes, UInt64 expected_ca if (remaining != 0) throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector bitmap has {} trailing bytes", remaining); - if (running_cardinality != expected_cardinality) + if (expected_cardinality.has_value() && running_cardinality != *expected_cardinality) throw Exception( ErrorCodes::BAD_ARGUMENTS, "Deletion vector cardinality {} does not match deserialized row count {}", - expected_cardinality, + *expected_cardinality, running_cardinality); + + return positions; +} + +static void deserializeRoaringPositionBitmap(std::string_view bytes, UInt64 expected_cardinality, ColumnUInt64 & positions) +{ + auto decoded = deserializeRoaringPositionBitmap(bytes, std::optional{expected_cardinality}); + for (UInt64 position : decoded) + positions.insertValue(position); } std::string_view extractDeletionVectorPayload(std::string_view blob) @@ -733,73 +840,9 @@ void deserializeDeletionVectorV1(std::string_view blob, UInt64 expected_cardinal deserializeRoaringPositionBitmap(extractDeletionVectorPayload(blob), expected_cardinality, positions); } -NamesAndTypesList getPuffinMetadataSchema() -{ - return { - {"blob_type", std::make_shared()}, - {"snapshot_id", std::make_shared()}, - {"sequence_number", std::make_shared()}, - {"fields", std::make_shared(std::make_shared())}, - {"offset", std::make_shared()}, - {"length", std::make_shared()}, - {"compression_codec", std::make_shared()}, - {"properties", std::make_shared(std::make_shared(), std::make_shared())}, - }; -} - -NamesAndTypesList getPuffinSchema() -{ - return { - {"referenced_data_file", std::make_shared()}, - {"deleted_rows", std::make_shared(std::make_shared())}, - }; -} - -void checkPuffinFormatHeader(const Block & header, const NamesAndTypesList & expected_schema, const char * format_name) -{ - std::unordered_map name_to_type; - for (const auto & [name, type] : expected_schema) - name_to_type[name] = type; - - String allowed_columns; - for (const auto & [name, type] : expected_schema) - { - if (!allowed_columns.empty()) - allowed_columns += ", "; - allowed_columns += name; - } - - for (const auto & [name, type] : header.getNamesAndTypes()) - { - auto it = name_to_type.find(name); - if (it == name_to_type.end()) - throw Exception( - ErrorCodes::BAD_ARGUMENTS, - "Unexpected column: {}. {} format allows only the next columns: {}", - name, - format_name, - allowed_columns); - - if (!it->second->equals(*type)) - throw Exception( - ErrorCodes::BAD_ARGUMENTS, - "Unexpected type {} for column {}. Expected type: {}", - type->getName(), - name, - it->second->getName()); - } -} - -void checkPuffinMetadataHeader(const Block & header) -{ - checkPuffinFormatHeader(header, getPuffinMetadataSchema(), "PuffinMetadata"); -} - -void checkPuffinHeader(const Block & header) +std::vector readPuffinFooterFromSeekable(SeekableReadBuffer & seekable, size_t file_size) { - checkPuffinFormatHeader(header, getPuffinSchema(), "Puffin"); -} - + return readPuffinFooterFromSeekableImpl(seekable, file_size); } PuffinMetadataInputFormat::PuffinMetadataInputFormat(ReadBuffer & buf, SharedHeader header_, const FormatSettings & format_settings_) @@ -906,29 +949,19 @@ Chunk PuffinInputFormat::read() const auto & blob = footer.blobs[blob_index++]; if (blob.type != PUFFIN_DELETION_VECTOR_BLOB_TYPE) - throw Exception( - ErrorCodes::BAD_ARGUMENTS, - "Puffin blob {}: unexpected blob type '{}', only '{}' is supported", - current_blob_index, - blob.type, - PUFFIN_DELETION_VECTOR_BLOB_TYPE); + continue; + const UInt64 expected_cardinality = requireDeletionVectorV1Properties(blob, current_blob_index); const auto & referenced_data_file = blob.properties.at("referenced-data-file"); - UInt64 expected_cardinality = 0; - if (!tryParse(expected_cardinality, blob.properties.at("cardinality"))) - throw Exception( - ErrorCodes::BAD_ARGUMENTS, - "Puffin blob {}: deletion-vector-v1 property 'cardinality' must be an unsigned integer", - current_blob_index); - auto col_file = ColumnString::create(); col_file->insertData(referenced_data_file.data(), referenced_data_file.size()); MutableColumnPtr col_rows; if (need_deleted_rows) { - const String blob_data = readDeletionVectorBlobBytes(blob, *in, footer.data, seekable_read); + const String blob_data + = readDeletionVectorBlobBytes(blob, *in, footer.data, seekable_read, expected_cardinality); auto col_rows_data = ColumnUInt64::create(); deserializeDeletionVectorV1(blob_data, expected_cardinality, *col_rows_data); @@ -1002,9 +1035,7 @@ void registerInputFormatPuffin(FormatFactory & factory) ## Description {#description} Special input format for reading [Apache Iceberg Puffin](https://iceberg.apache.org/puffin-spec/) file footer metadata. -It outputs one row per blob entry from the footer `BlobMetadata` list. - -`deletion-vector-v1` is the only supported blob type: a file containing any other blob type (for example `apache-datasketches-theta-v1`) is rejected. +It outputs one row per blob entry from the footer `BlobMetadata` list, including non-deletion-vector types (for example `apache-datasketches-theta-v1`). Full deletion-vector property validation applies only to `deletion-vector-v1` entries. Fixed output columns: - `blob_type` (`String`) - blob type, for example `deletion-vector-v1` @@ -1047,16 +1078,16 @@ Pair with the `Puffin` format to read `deletion-vector-v1` blob payloads. Input format for reading [Apache Iceberg Puffin](https://iceberg.apache.org/puffin-spec/) files. -The format exposes deleted row positions from `deletion-vector-v1` blobs. It is the only supported blob type: a file containing any other blob type (for example `apache-datasketches-theta-v1`) is rejected. +The format exposes deleted row positions from `deletion-vector-v1` blobs. Other blob types (for example `apache-datasketches-theta-v1`) are skipped. If a puffin file contains multiple `deletion-vector-v1` blobs, the format outputs one row per such blob. Fixed output columns: - `referenced_data_file` (`String`) - location of the data file the deletion vector applies to (`referenced-data-file` blob property) - `deleted_rows` (`Array(UInt64)`) - 64-bit row positions deleted according to the deletion vector roaring bitmap -Deletion vectors whose declared `cardinality` exceeds an absolute materialization ceiling are rejected when `deleted_rows` is requested. Footer `deletion-vector-v1` properties (including that `cardinality` parses as an unsigned integer) are always validated. Selecting only `referenced_data_file` skips on-disk payload I/O and therefore also skips envelope, CRC, roaring deserialize, and the materialization ceiling — intentionally, so a path-only projection does not read up to the blob-size cap. +Deletion vectors whose declared `cardinality` exceeds an absolute materialization ceiling are rejected when `deleted_rows` is requested, **before** envelope peek or full blob allocation (same fail-closed order as the Iceberg deletion-vector reader). Footer `deletion-vector-v1` properties are always validated: `cardinality` must parse as an unsigned integer, `snapshot-id` / `sequence-number` must be `-1`, and `fields` must be either empty or the singleton Iceberg reserved `_pos` id (`2147483645`) that Spark writes for file-scoped DVs — other `fields` lists (column-scoped DVs) are rejected. Selecting only `referenced_data_file` skips on-disk payload I/O and therefore also skips envelope, CRC, roaring deserialize, and the materialization ceiling — intentionally, so a path-only projection does not read up to the blob-size cap. -On-disk `deletion-vector-v1` blob length is bounded by an absolute ceiling (aligned with Iceberg's 2 GiB content-size check). When `deleted_rows` is requested, the reader peeks the envelope header (combined length and magic) before allocating the full payload; CRC is verified after the bounded read. +On-disk `deletion-vector-v1` blob length is bounded by an absolute ceiling (aligned with Iceberg's 2 GiB content-size check). When `deleted_rows` is requested and cardinality is within the materialization ceiling, the reader peeks the envelope header (combined length and magic) before allocating the full payload; CRC is verified after the bounded read. LZ4-compressed and uncompressed puffin footers are supported. Footer payload size (and declared LZ4 content size) is bounded by a compression ratio where applicable and an absolute ceiling; oversized footers are rejected before allocation. diff --git a/src/Processors/Formats/Impl/PuffinBlockInputFormat.h b/src/Processors/Formats/Impl/PuffinBlockInputFormat.h index 203dcd368544..251e96ef60f5 100644 --- a/src/Processors/Formats/Impl/PuffinBlockInputFormat.h +++ b/src/Processors/Formats/Impl/PuffinBlockInputFormat.h @@ -1,10 +1,16 @@ #pragma once +#include #include +#include #include #include +#include +#include +#include + namespace DB { @@ -26,6 +32,20 @@ struct PuffinFooter std::vector data; }; +/// Shared with the Iceberg deletion-vector loader (seekable object-storage path). +std::vector readPuffinFooterFromSeekable(SeekableReadBuffer & seekable, size_t file_size); + +/// Shared deletion-vector-v1 payload helpers (also used by `PuffinDeletionVectorReader`). +std::string_view extractDeletionVectorPayload(std::string_view blob); +std::vector deserializeRoaringPositionBitmap( + std::string_view bytes, std::optional expected_cardinality = std::nullopt); +void deserializeDeletionVectorV1(std::string_view blob, UInt64 expected_cardinality, ColumnUInt64 & positions); + +/// Validate deletion-vector-v1 footer identity (`snapshot-id` / `sequence-number` / `fields` / +/// required string properties). Returns parsed `cardinality`. Used by the SQL `Puffin` path and +/// by `bindDeletionVectorBlob`. +UInt64 requireDeletionVectorV1Properties(const PuffinBlob & blob, size_t blob_index); + class PuffinMetadataInputFormat : public IInputFormat { public: diff --git a/src/Processors/tests/gtest_parquet_row_group_global_offsets.cpp b/src/Processors/tests/gtest_parquet_row_group_global_offsets.cpp new file mode 100644 index 000000000000..664a8a43defa --- /dev/null +++ b/src/Processors/tests/gtest_parquet_row_group_global_offsets.cpp @@ -0,0 +1,85 @@ +#include + +#include + +#if USE_PARQUET + +#include +#include +#include + +#include + +using namespace DB; +using namespace DB::Parquet; + +namespace DB::ErrorCodes +{ +extern const int INCORRECT_DATA; +} + +namespace +{ + +parq::FileMetaData makeFileMetaData(Int64 file_num_rows, const std::vector & row_group_num_rows) +{ + parq::FileMetaData meta; + meta.__set_num_rows(file_num_rows); + meta.row_groups.reserve(row_group_num_rows.size()); + for (Int64 num_rows : row_group_num_rows) + { + parq::RowGroup row_group; + row_group.__set_num_rows(num_rows); + meta.row_groups.push_back(std::move(row_group)); + } + return meta; +} + +} + +TEST(ParquetRowGroupGlobalOffsets, BuildsPrefixOffsets) +{ + const auto offsets = buildRowGroupGlobalOffsets(makeFileMetaData(30, {10, 0, 20})); + ASSERT_EQ(offsets.size(), 4u); + EXPECT_EQ(offsets[0], 0u); + EXPECT_EQ(offsets[1], 10u); + EXPECT_EQ(offsets[2], 10u); + EXPECT_EQ(offsets[3], 30u); +} + +TEST(ParquetRowGroupGlobalOffsets, RejectsNegativeFileRowCount) +{ + EXPECT_THROW(buildRowGroupGlobalOffsets(makeFileMetaData(-1, {})), Exception); +} + +TEST(ParquetRowGroupGlobalOffsets, RejectsNegativeRowGroupRowCount) +{ + EXPECT_THROW(buildRowGroupGlobalOffsets(makeFileMetaData(10, {10, -1})), Exception); +} + +TEST(ParquetRowGroupGlobalOffsets, ToleratesMismatchWithFileNumRows) +{ + /// Offsets follow row-group counts even when FileMetaData.num_rows is stale/wrong. + const auto offsets = buildRowGroupGlobalOffsets(makeFileMetaData(11, {10, 0})); + ASSERT_EQ(offsets.size(), 3u); + EXPECT_EQ(offsets[0], 0u); + EXPECT_EQ(offsets[1], 10u); + EXPECT_EQ(offsets[2], 10u); +} + +TEST(ParquetRowGroupGlobalOffsets, RejectsOverflowingCumulativeCounts) +{ + constexpr Int64 max_rows = std::numeric_limits::max(); + /// Two Int64::max values fit in UInt64; the third overflows checked addition. + try + { + buildRowGroupGlobalOffsets(makeFileMetaData(max_rows, {max_rows, max_rows, max_rows})); + FAIL() << "Expected INCORRECT_DATA"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::INCORRECT_DATA); + } +} + +#endif diff --git a/src/Processors/tests/gtest_parquet_v3_need_only_count_buckets.cpp b/src/Processors/tests/gtest_parquet_v3_need_only_count_buckets.cpp new file mode 100644 index 000000000000..ea0f28a5c81a --- /dev/null +++ b/src/Processors/tests/gtest_parquet_v3_need_only_count_buckets.cpp @@ -0,0 +1,191 @@ +#include + +#include + +#if USE_PARQUET + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +using namespace DB; + +namespace DB::ErrorCodes +{ +extern const int LOGICAL_ERROR; +} + +namespace +{ + +void writeMultiRowGroupParquet(const String & path, size_t rows_per_group, size_t num_groups) +{ + FormatSettings format_settings; + format_settings.parquet.row_group_rows = rows_per_group; + format_settings.parquet.parallel_encoding = false; + + Block header; + header.insert(ColumnWithTypeAndName(std::make_shared(), "x")); + + Chunks chunks; + for (size_t group = 0; group < num_groups; ++group) + { + auto column = ColumnUInt64::create(); + for (size_t i = 0; i < rows_per_group; ++i) + column->insert(group * rows_per_group + i); + chunks.emplace_back(Columns{std::move(column)}, rows_per_group); + } + + auto source = std::make_shared(std::make_shared(header), std::move(chunks)); + QueryPipelineBuilder pipeline_builder; + pipeline_builder.init(Pipe(source)); + auto pipeline = QueryPipelineBuilder::getPipeline(std::move(pipeline_builder)); + + WriteBufferFromFile write_buffer(path); + auto output = std::make_shared(write_buffer, pipeline.getSharedHeader(), format_settings, nullptr); + pipeline.complete(output); + CompletedPipelineExecutor executor(pipeline); + executor.execute(); + output->finalize(); + write_buffer.finalize(); +} + +size_t readNeedOnlyCountTotal( + ReadBuffer & in, + const SharedHeader & header, + const FormatSettings & format_settings, + FormatParserSharedResourcesPtr parser_shared_resources, + const FileBucketInfoPtr & buckets) +{ + auto input = std::make_shared( + in, + header, + format_settings, + parser_shared_resources, + std::make_shared(), + /*min_bytes_for_seek=*/ 1024); + if (buckets) + input->setBucketsToRead(buckets); + input->needOnlyCount(); + + size_t total = 0; + while (true) + { + Chunk chunk = input->generate(); + if (!chunk) + break; + total += chunk.getNumRows(); + } + return total; +} + +std::vector> readNeedOnlyCountOffsetsAndRows( + ReadBuffer & in, + const SharedHeader & header, + const FormatSettings & format_settings, + FormatParserSharedResourcesPtr parser_shared_resources, + const FileBucketInfoPtr & buckets) +{ + auto input = std::make_shared( + in, + header, + format_settings, + parser_shared_resources, + std::make_shared(), + /*min_bytes_for_seek=*/ 1024); + if (buckets) + input->setBucketsToRead(buckets); + input->needOnlyCount(); + + std::vector> result; + while (true) + { + Chunk chunk = input->generate(); + if (!chunk) + break; + auto info = chunk.getChunkInfos().get(); + if (!info) + throw Exception(ErrorCodes::LOGICAL_ERROR, "Expected ChunkInfoRowNumbers on need_only_count chunk"); + result.emplace_back(info->row_num_offset, chunk.getNumRows()); + } + return result; +} + +} + +TEST(ParquetV3NeedOnlyCountBuckets, CountsOnlyAssignedRowGroups) +{ + tryRegisterFormats(); + const auto context = getContext().context; + + Poco::TemporaryFile temp_file; + const String path = temp_file.path(); + constexpr size_t rows_per_group = 10; + constexpr size_t num_groups = 3; + writeMultiRowGroupParquet(path, rows_per_group, num_groups); + + Block header; + header.insert(ColumnWithTypeAndName(std::make_shared(), "x")); + auto shared_header = std::make_shared(header); + FormatSettings format_settings; + auto parser_shared_resources = FormatParserSharedResources::singleThreaded(context->getSettingsRef()); + + { + ReadBufferFromFile in(path); + EXPECT_EQ( + readNeedOnlyCountTotal(in, shared_header, format_settings, parser_shared_resources, /*buckets=*/ nullptr), + rows_per_group * num_groups); + } + + { + /// Unbucketed need-only-count uses `FileMetaData.num_rows` as a single whole-file span. + ReadBufferFromFile in(path); + const auto offsets_and_rows + = readNeedOnlyCountOffsetsAndRows(in, shared_header, format_settings, parser_shared_resources, /*buckets=*/ nullptr); + ASSERT_EQ(offsets_and_rows.size(), 1u); + EXPECT_EQ(offsets_and_rows[0].first, 0u); + EXPECT_EQ(offsets_and_rows[0].second, rows_per_group * num_groups); + } + + { + ReadBufferFromFile in(path); + auto buckets = std::make_shared(std::vector{1}); + EXPECT_EQ(readNeedOnlyCountTotal(in, shared_header, format_settings, parser_shared_resources, buckets), rows_per_group); + } + + { + /// Bucketed tasks must count only assigned row groups (not the whole file) and keep + /// file-global row offsets for downstream row-number consumers. + ReadBufferFromFile in(path); + auto buckets = std::make_shared(std::vector{0, 2}); + const auto offsets_and_rows + = readNeedOnlyCountOffsetsAndRows(in, shared_header, format_settings, parser_shared_resources, buckets); + ASSERT_EQ(offsets_and_rows.size(), 2u); + EXPECT_EQ(offsets_and_rows[0].first, 0u); + EXPECT_EQ(offsets_and_rows[0].second, rows_per_group); + EXPECT_EQ(offsets_and_rows[1].first, 2 * rows_per_group); + EXPECT_EQ(offsets_and_rows[1].second, rows_per_group); + } + + { + ReadBufferFromFile in(path); + auto buckets = std::make_shared(std::vector{}); + EXPECT_EQ(readNeedOnlyCountTotal(in, shared_header, format_settings, parser_shared_resources, buckets), 0u); + } +} + +#endif diff --git a/src/Storages/ObjectStorage/DataLakes/Common/AvroForIcebergDeserializer.cpp b/src/Storages/ObjectStorage/DataLakes/Common/AvroForIcebergDeserializer.cpp index 1500ffd86242..92bbeca6e9ee 100644 --- a/src/Storages/ObjectStorage/DataLakes/Common/AvroForIcebergDeserializer.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Common/AvroForIcebergDeserializer.cpp @@ -5,6 +5,7 @@ #include #include #include +#include #include #include #include @@ -16,6 +17,7 @@ #include #include #include +#include #include namespace DB::ErrorCodes @@ -252,6 +254,22 @@ ParsedManifestFileEntryPtr AvroForIcebergDeserializer::createParsedManifestFileE const auto record_count = getValueFromRowByName(row_index, c_data_file_record_count, TypeIndex::Int64).safeGet(); const auto file_size_in_bytes = getValueFromRowByName(row_index, c_data_file_file_size_in_bytes, TypeIndex::Int64).safeGet(); + std::optional content_offset; + if (hasPath(c_data_file_content_offset)) + { + const auto content_offset_value = getValueFromRowByName(row_index, c_data_file_content_offset); + if (!content_offset_value.isNull()) + content_offset = content_offset_value.safeGet(); + } + + std::optional content_size_in_bytes; + if (hasPath(c_data_file_content_size_in_bytes)) + { + const auto content_size_value = getValueFromRowByName(row_index, c_data_file_content_size_in_bytes); + if (!content_size_value.isNull()) + content_size_in_bytes = content_size_value.safeGet(); + } + switch (content_type) { case FileContentType::DATA: { @@ -275,6 +293,7 @@ ParsedManifestFileEntryPtr AvroForIcebergDeserializer::createParsedManifestFileE } case FileContentType::POSITION_DELETE: { /// reference_file_path can be absent in schema for some reason, though it is present in specification: https://iceberg.apache.org/spec/#manifests + const bool is_puffin = Poco::toLower(file_format) == "puffin"; std::optional lower_reference_data_file_path; std::optional upper_reference_data_file_path; bool bounds_set_by_referenced_data_file = false; @@ -290,7 +309,9 @@ ParsedManifestFileEntryPtr AvroForIcebergDeserializer::createParsedManifestFileE bounds_set_by_referenced_data_file = true; } } - if (!bounds_set_by_referenced_data_file) + /// Parquet position deletes may fall back to file-path column bounds. Puffin deletion + /// vectors must use the dedicated referenced_data_file field only. + if (!bounds_set_by_referenced_data_file && !is_puffin) { if (auto it = value_for_bounds.find(IcebergPositionDeleteTransform::data_file_path_column_field_id); it != value_for_bounds.end()) @@ -302,6 +323,20 @@ ParsedManifestFileEntryPtr AvroForIcebergDeserializer::createParsedManifestFileE upper_reference_data_file_path.emplace(Iceberg::IcebergPathFromMetadata::deserialize(upper.safeGet())); } } + + if (is_puffin) + { + if (!content_offset.has_value() || !content_size_in_bytes.has_value()) + { + throw Exception( + DB::ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION, + "Puffin deletion vector entry in manifest file '{}' is missing content_offset or content_size_in_bytes", + manifest_file_path); + } + requireDirectReferencedDataFileForPuffinDeletionVector( + bounds_set_by_referenced_data_file, lower_reference_data_file_path, manifest_file_path); + } + return std::make_shared( FileContentType::POSITION_DELETE, file_path_key, @@ -318,7 +353,9 @@ ParsedManifestFileEntryPtr AvroForIcebergDeserializer::createParsedManifestFileE /*equality_ids*/ std::nullopt, /*sort_order_id = */ std::nullopt, record_count, - file_size_in_bytes); + file_size_in_bytes, + content_offset, + content_size_in_bytes); } case FileContentType::EQUALITY_DELETE: { std::vector equality_ids; diff --git a/src/Storages/ObjectStorage/DataLakes/DataLakeObjectMetadata.h b/src/Storages/ObjectStorage/DataLakes/DataLakeObjectMetadata.h index fcc2c0495e18..566b1965e0df 100644 --- a/src/Storages/ObjectStorage/DataLakes/DataLakeObjectMetadata.h +++ b/src/Storages/ObjectStorage/DataLakes/DataLakeObjectMetadata.h @@ -1,6 +1,8 @@ #pragma once #include +#include + namespace DB { template @@ -17,4 +19,11 @@ struct DataLakeObjectMetadata ExcludedRowsPtr excluded_rows; }; +/// True when a deletion vector (or similar) will filter rows via DeletionVectorTransform. +/// Count-from-files cache must skip these objects: the cache key is data-file identity only, +/// while excluded_rows can change independently (Iceberg puffin DVs, DeltaLake selection vectors). +/// Cluster task serialization must also fail closed when these cannot be carried on the wire. +bool hasNonEmptyExcludedRows(const DataLakeObjectMetadata & metadata); +bool hasNonEmptyExcludedRows(const std::optional & metadata); + } diff --git a/src/Storages/ObjectStorage/DataLakes/DeletionVectorTransform.cpp b/src/Storages/ObjectStorage/DataLakes/DeletionVectorTransform.cpp index 3fd353ef99a1..50c6d6db16ad 100644 --- a/src/Storages/ObjectStorage/DataLakes/DeletionVectorTransform.cpp +++ b/src/Storages/ObjectStorage/DataLakes/DeletionVectorTransform.cpp @@ -3,6 +3,7 @@ #include #include #include +#include namespace DB::ErrorCodes @@ -12,6 +13,105 @@ namespace DB::ErrorCodes namespace DB { +namespace +{ + +bool isConstCountOnlyChunk(const Chunk & chunk, const ChunkInfoRowNumbers & chunk_info) +{ + /// need_only_count emits ColumnConst defaults with no prior filter. Dense filtering is still + /// required when real column values or an applied_filter (PREWHERE / prior filters) are present. + if (chunk_info.applied_filter.has_value()) + return false; + + for (const auto & column : chunk.getColumns()) + { + if (!isColumnConst(*column)) + return false; + } + return true; +} + +void transformConstCountOnlyChunk(Chunk & chunk, ChunkInfoRowNumbers & chunk_info, const DataLakeObjectMetadata::ExcludedRows & excluded_rows) +{ + const size_t num_rows_before = chunk.getNumRows(); + size_t range_end = 0; + if (common::addOverflow(chunk_info.row_num_offset, num_rows_before, range_end)) + { + throw Exception( + ErrorCodes::LOGICAL_ERROR, + "Deletion vector count range overflows size_t: offset {} num_rows {}", + chunk_info.row_num_offset, + num_rows_before); + } + + const UInt64 deleted = excluded_rows.rb_range_cardinality(chunk_info.row_num_offset, range_end); + if (deleted > num_rows_before) + { + throw Exception( + ErrorCodes::LOGICAL_ERROR, + "Deletion vector reports {} deletes in a count chunk of {} rows (offset {})", + deleted, + num_rows_before, + chunk_info.row_num_offset); + } + + const size_t num_rows_after = num_rows_before - static_cast(deleted); + if (num_rows_after == num_rows_before) + return; + + auto columns = chunk.detachColumns(); + for (auto & column : columns) + column = column->cloneResized(num_rows_after); + + /// Same invariant as the dense path: after shrinking, `applied_filter` maps dense indices back + /// to file rows (`row_num_offset` + index of the i-th set bit). Without this, a later + /// row-number consumer (e.g. another `DeletionVectorTransform` from parquet position deletes) + /// would treat the shrunk const chunk as consecutive file rows `[offset, offset + num_rows)`. + IColumn::Filter filter(num_rows_before, 1); + DataLakeObjectMetadata::ExcludedRows deleted_in_range; + const UInt64 ranged = excluded_rows.rb_range(chunk_info.row_num_offset, range_end, deleted_in_range); + if (ranged != deleted) + { + throw Exception( + ErrorCodes::LOGICAL_ERROR, + "Deletion vector range cardinality {} disagrees with rb_range size {} (offset {} num_rows {})", + deleted, + ranged, + chunk_info.row_num_offset, + num_rows_before); + } + + PaddedPODArray deleted_positions; + deleted_in_range.rb_to_array(deleted_positions); + for (size_t position : deleted_positions) + { + if (position < chunk_info.row_num_offset || position >= range_end) + { + throw Exception( + ErrorCodes::LOGICAL_ERROR, + "Deletion vector position {} outside count range [{}, {})", + position, + chunk_info.row_num_offset, + range_end); + } + filter[position - chunk_info.row_num_offset] = 0; + } + + chunk_info.applied_filter.emplace(std::move(filter)); + chunk.setColumns(std::move(columns), num_rows_after); +} + +} + +bool hasNonEmptyExcludedRows(const DataLakeObjectMetadata & metadata) +{ + return metadata.excluded_rows && metadata.excluded_rows->size() > 0; +} + +bool hasNonEmptyExcludedRows(const std::optional & metadata) +{ + return metadata.has_value() && hasNonEmptyExcludedRows(*metadata); +} DeletionVectorTransform::DeletionVectorTransform( const DB::SharedHeader & header_, @@ -32,6 +132,12 @@ void DeletionVectorTransform::transform(DB::Chunk & chunk, const ExcludedRows & if (!chunk_info) throw DB::Exception(DB::ErrorCodes::LOGICAL_ERROR, "ChunkInfoRowNumbers does not exist"); + if (isConstCountOnlyChunk(chunk, *chunk_info)) + { + transformConstCountOnlyChunk(chunk, *chunk_info, excluded_rows); + return; + } + const size_t num_rows_before = chunk.getNumRows(); size_t num_rows_after = num_rows_before; size_t idx_in_chunk = 0; diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/Constant.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/Constant.h index 1f2443a92460..cd7c9f29d7e3 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/Constant.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/Constant.h @@ -177,6 +177,8 @@ DEFINE_ICEBERG_FIELD_COMPOUND(data_file, null_value_counts); DEFINE_ICEBERG_FIELD_COMPOUND(data_file, lower_bounds); DEFINE_ICEBERG_FIELD_COMPOUND(data_file, upper_bounds); DEFINE_ICEBERG_FIELD_COMPOUND(data_file, referenced_data_file); +DEFINE_ICEBERG_FIELD_COMPOUND(data_file, content_offset); +DEFINE_ICEBERG_FIELD_COMPOUND(data_file, content_size_in_bytes); DEFINE_ICEBERG_FIELD_COMPOUND(data_file, sort_order_id); DEFINE_ICEBERG_FIELD_COMPOUND(data_file, record_count); DEFINE_ICEBERG_FIELD_COMPOUND(data_file, file_size_in_bytes); diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDataObjectInfo.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDataObjectInfo.cpp index 4e161ca8c863..6e99248bcba4 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDataObjectInfo.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDataObjectInfo.cpp @@ -15,6 +15,7 @@ #include #include +#include namespace DB::ErrorCodes { @@ -25,6 +26,23 @@ extern const int UNKNOWN_PROTOCOL; using namespace DB::Iceberg; +namespace DB::Iceberg +{ + +void requireParquetDataFileForRowDeletes(const String & file_format, std::string_view feature_name) +{ + if (Poco::toUpper(file_format) != "PARQUET") + { + throw Exception( + DB::ErrorCodes::NOT_IMPLEMENTED, + "{} are only supported for data files of Parquet format in Iceberg, but got {}", + feature_name, + file_format); + } +} + +} + namespace DB { @@ -97,13 +115,7 @@ std::shared_ptr IcebergDataObjectInfo::getPositionDeleteTransf void IcebergDataObjectInfo::addPositionDeleteObject(Iceberg::ProcessedManifestFileEntryPtr position_delete_object, const String & resolved_storage_path) { - if (Poco::toUpper(info.file_format) != "PARQUET") - { - throw Exception( - ErrorCodes::NOT_IMPLEMENTED, - "Position deletes are only supported for data files of Parquet format in Iceberg, but got {}", - info.file_format); - } + Iceberg::requireParquetDataFileForRowDeletes(info.file_format, "Position deletes"); info.position_deletes_objects.emplace_back( resolved_storage_path, position_delete_object->parsed_entry->file_format, std::nullopt, position_delete_object->sequence_number); @@ -118,6 +130,26 @@ void IcebergDataObjectInfo::addEqualityDeleteObject(const Iceberg::ProcessedMani equality_delete_object->resolved_schema_id); } +ObjectInfoPtr IcebergDataObjectInfo::clone() const +{ + auto result = std::make_shared(relative_path_with_metadata, info); + result->data_lake_metadata = data_lake_metadata; + result->file_bucket_info = file_bucket_info; + return result; +} + +bool hasIcebergEqualityDeletes(const ObjectInfoPtr & object_info) +{ + const auto * iceberg = dynamic_cast(object_info.get()); + return iceberg && !iceberg->info.equality_deletes_objects.empty(); +} + +bool hasIcebergPositionDeletes(const ObjectInfoPtr & object_info) +{ + const auto * iceberg = dynamic_cast(object_info.get()); + return iceberg && !iceberg->info.position_deletes_objects.empty(); +} + #endif void IcebergObjectSerializableInfo::serializeForClusterFunctionProtocol(WriteBuffer & out, size_t protocol_version) const diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDataObjectInfo.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDataObjectInfo.h index 6763d1013201..9e646eedd52c 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDataObjectInfo.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDataObjectInfo.h @@ -11,12 +11,18 @@ #include #include +#include + namespace DB::Iceberg { String computePartitionId(const Row & partition_key_value); +/// Position deletes and deletion vectors need file-relative row numbers (`ChunkInfoRowNumbers`), +/// which Parquet readers emit. Reject other data-file formats early with NOT_IMPLEMENTED. +void requireParquetDataFileForRowDeletes(const String & file_format, std::string_view feature_name); + struct IcebergObjectSerializableInfo { @@ -84,10 +90,18 @@ struct IcebergDataObjectInfo : public ObjectInfo, std::enable_shared_from_this; + +/// Equality deletes need real column values — disable need_only_count / count-from-files cache. +bool hasIcebergEqualityDeletes(const ObjectInfoPtr & object_info); +/// Position-delete attachment changes independently of data-file identity — skip count-from-files cache. +bool hasIcebergPositionDeletes(const ObjectInfoPtr & object_info); } #endif diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDeletionVector.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDeletionVector.cpp new file mode 100644 index 000000000000..2ec418713090 --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDeletionVector.cpp @@ -0,0 +1,325 @@ +#include "config.h" + +#if USE_AVRO + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace DB +{ + +namespace ErrorCodes +{ + extern const int ICEBERG_SPECIFICATION_VIOLATION; + extern const int BAD_ARGUMENTS; + extern const int LOGICAL_ERROR; +} + +namespace Setting +{ +extern const SettingsBool use_puffin_files_cache; +} + +} + +namespace DB::Iceberg +{ + +void validateDeletionVectorPositionsAgainstDataFile( + std::span deleted_positions, + UInt64 expected_cardinality, + Int64 data_file_record_count) +{ + if (data_file_record_count < 0) + { + throw Exception( + ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION, + "Data file record_count {} must be non-negative", + data_file_record_count); + } + + const UInt64 data_file_rows = static_cast(data_file_record_count); + + if (expected_cardinality > data_file_rows) + { + throw Exception( + ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION, + "Deletion vector cardinality {} exceeds data file record_count {}", + expected_cardinality, + data_file_record_count); + } + + for (UInt64 position : deleted_positions) + { + if (position >= data_file_rows) + { + throw Exception( + ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION, + "Deletion vector position {} is out of range for data file record_count {}", + position, + data_file_record_count); + } + } +} + +namespace +{ + +using FooterBlobsPtr = PuffinFilesCache::FooterBlobsPtr; + +FooterBlobsPtr readFooterBlobs( + ObjectStoragePtr object_storage, + const String & puffin_path, + ContextPtr context, + LoggerPtr log, + bool disable_filesystem_cache) +{ + RelativePathWithMetadata puffin_object{puffin_path}; + auto read_settings = context->getReadSettings(); + if (disable_filesystem_cache) + read_settings.enable_filesystem_cache = false; + + auto read_buffer = createReadBuffer(puffin_object, object_storage, context, log, read_settings); + + auto * seekable = dynamic_cast(read_buffer.get()); + if (!seekable) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Puffin deletion vector read requires a seekable buffer"); + + auto file_size = tryGetFileSizeFromReadBuffer(*read_buffer); + if (!file_size) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Cannot determine Puffin file size for '{}'", puffin_path); + + return std::make_shared>(readPuffinFooterBlobsFromSeekable(*seekable, *file_size)); +} + +DataLakeObjectMetadata::ExcludedRowsPtr loadDeletionVectorUncached( + ObjectStoragePtr object_storage, + const String & puffin_path, + Int64 content_offset, + Int64 content_size_in_bytes, + const IcebergPathFromMetadata & expected_data_file, + UInt64 expected_cardinality, + Int64 data_file_record_count, + ContextPtr context, + LoggerPtr log, + bool disable_filesystem_cache, + FooterBlobsPtr preloaded_footer) +{ + RelativePathWithMetadata puffin_object{puffin_path}; + auto read_settings = context->getReadSettings(); + if (disable_filesystem_cache) + read_settings.enable_filesystem_cache = false; + + auto read_buffer = createReadBuffer(puffin_object, object_storage, context, log, read_settings); + + auto * seekable = dynamic_cast(read_buffer.get()); + if (!seekable) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Puffin deletion vector read requires a seekable buffer"); + + auto file_size = tryGetFileSizeFromReadBuffer(*read_buffer); + if (!file_size) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Cannot determine Puffin file size for '{}'", puffin_path); + + FooterBlobsPtr footer_owner = preloaded_footer; + if (!footer_owner) + footer_owner = std::make_shared>(readPuffinFooterBlobsFromSeekable(*seekable, *file_size)); + + bindDeletionVectorBlob( + *footer_owner, + content_offset, + content_size_in_bytes, + expected_data_file.serialize(), + expected_cardinality); + + auto deleted_positions = readDeletionVectorFromPuffin( + *read_buffer, content_offset, content_size_in_bytes, expected_cardinality); + + validateDeletionVectorPositionsAgainstDataFile(deleted_positions, expected_cardinality, data_file_record_count); + + if (deleted_positions.empty()) + return nullptr; + + auto bitmap = std::make_shared(); + for (UInt64 position : deleted_positions) + bitmap->add(static_cast(position)); + + LOG_DEBUG( + log, + "Loaded deletion vector from puffin file '{}' for data file '{}': {} deleted rows", + puffin_path, + expected_data_file.serialize(), + deleted_positions.size()); + + return bitmap; +} + +} + +DataLakeObjectMetadata::ExcludedRowsPtr loadDeletionVector( + ObjectStoragePtr object_storage, + const String & puffin_path, + Int64 content_offset, + Int64 content_size_in_bytes, + const IcebergPathFromMetadata & expected_data_file, + const std::optional & referenced_data_file, + Int64 expected_cardinality, + Int64 data_file_record_count, + ContextPtr context, + LoggerPtr log) +{ + if (referenced_data_file.has_value() && referenced_data_file.value() != expected_data_file) + { + throw Exception( + ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION, + "Deletion vector referenced_data_file '{}' does not match data file '{}'", + referenced_data_file->serialize(), + expected_data_file.serialize()); + } + + if (expected_cardinality < 0) + { + throw Exception( + ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION, + "Deletion vector record_count {} must be non-negative", + expected_cardinality); + } + + if (data_file_record_count < 0) + { + throw Exception( + ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION, + "Data file record_count {} must be non-negative", + data_file_record_count); + } + + const UInt64 expected_cardinality_u64 = static_cast(expected_cardinality); + + /// Fail closed before I/O when declared DV cardinality cannot fit in the data file. + if (expected_cardinality_u64 > static_cast(data_file_record_count)) + { + throw Exception( + ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION, + "Deletion vector cardinality {} exceeds data file record_count {}", + expected_cardinality, + data_file_record_count); + } + + const bool use_cache_setting = context->getSettingsRef()[Setting::use_puffin_files_cache]; + auto cache = use_cache_setting ? context->getPuffinFilesCache() : nullptr; + + /// puffin_files_cache_size=0 means the LRU accepts no entries ("disabled" in server settings). + /// Take the same uncached path as use_puffin_files_cache=0 so we keep filesystem cache and + /// skip etag HEAD / getOrSet (which would disable filesystem cache on the miss loader). + const bool use_cache = cache && cache->maxSizeInBytes() != 0; + if (!use_cache) + { + if (!use_cache_setting) + { + LOG_TRACE(log, "Not using Puffin files cache for '{}', because the setting use_puffin_files_cache is false", puffin_path); + } + else + { + LOG_TRACE( + log, + "Not using Puffin files cache for '{}', because puffin_files_cache_size is 0", + puffin_path); + } + return loadDeletionVectorUncached( + object_storage, + puffin_path, + content_offset, + content_size_in_bytes, + expected_data_file, + expected_cardinality_u64, + data_file_record_count, + context, + log, + false, + nullptr); + } + + RelativePathWithMetadata puffin_object{puffin_path}; + if (!puffin_object.metadata) + puffin_object.metadata = object_storage->getObjectMetadata(puffin_object.getPath(), /*with_tags=*/ false); + + if (puffin_object.metadata->etag.empty()) + { + LOG_TRACE( + log, + "Not using Puffin files cache for '{}', because etag is empty", + puffin_path); + return loadDeletionVectorUncached( + object_storage, + puffin_path, + content_offset, + content_size_in_bytes, + expected_data_file, + expected_cardinality_u64, + data_file_record_count, + context, + log, + false, + nullptr); + } + + const String storage_identity = PuffinFilesCache::makeStorageIdentity(*object_storage); + + auto footer_key = PuffinFilesCache::tryCreateFooterKey(storage_identity, puffin_path, puffin_object.metadata->etag); + auto cache_key = PuffinFilesCache::tryCreateKey( + storage_identity, + puffin_path, + puffin_object.metadata->etag, + content_offset, + content_size_in_bytes, + expected_data_file.serialize(), + expected_cardinality_u64, + static_cast(data_file_record_count)); + + /// Empty etag is the only reason tryCreate* returns nullopt; that case is handled above. + if (!footer_key || !cache_key) + { + throw Exception( + ErrorCodes::LOGICAL_ERROR, + "PuffinFilesCache::tryCreate* returned nullopt for non-empty etag on '{}'", + puffin_path); + } + + /// Footer is keyed by file identity only, so N DV slices in one coalesced Puffin share one parse. + /// Resolve the footer only on a deletion-vector cache miss (nested memo lookup). + return cache->getOrSetDeletionVector(*cache_key, [&]() + { + auto footer = cache->getOrSetFooter(*footer_key, [&]() + { + return readFooterBlobs(object_storage, puffin_path, context, log, /*disable_filesystem_cache=*/ true); + }); + + return loadDeletionVectorUncached( + object_storage, + puffin_path, + content_offset, + content_size_in_bytes, + expected_data_file, + expected_cardinality_u64, + data_file_record_count, + context, + log, + true, + footer); + }); +} + +} + +#endif diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDeletionVector.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDeletionVector.h new file mode 100644 index 000000000000..a0323730564d --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDeletionVector.h @@ -0,0 +1,38 @@ +#pragma once +#include "config.h" + +#if USE_AVRO + +#include +#include +#include +#include +#include + +#include +#include + +namespace DB::Iceberg +{ + +/// Reject DV positions outside `[0, data_file_record_count)` and cardinality that cannot fit. +void validateDeletionVectorPositionsAgainstDataFile( + std::span deleted_positions, + UInt64 expected_cardinality, + Int64 data_file_record_count); + +DataLakeObjectMetadata::ExcludedRowsPtr loadDeletionVector( + ObjectStoragePtr object_storage, + const String & puffin_path, + Int64 content_offset, + Int64 content_size_in_bytes, + const IcebergPathFromMetadata & expected_data_file, + const std::optional & referenced_data_file, + Int64 expected_cardinality, + Int64 data_file_record_count, + ContextPtr context, + LoggerPtr log); + +} + +#endif diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.cpp index 8fa5e76fe7a4..5147118d5c13 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.cpp @@ -43,6 +43,7 @@ #include #include #include +#include #include #include @@ -71,6 +72,7 @@ namespace DB namespace ErrorCodes { extern const int LOGICAL_ERROR; +extern const int ICEBERG_SPECIFICATION_VIOLATION; } namespace Setting { @@ -270,6 +272,7 @@ IcebergIterator::IcebergIterator( persistent_components_) , blocking_queue(100) , callback(std::move(callback_)) + , local_context(local_context_) , table_schema_id(table_snapshot_->schema_id) { auto delete_file = deletes_iterator.next(); @@ -279,15 +282,25 @@ IcebergIterator::IcebergIterator( { equality_deletes_files.emplace_back(std::move(delete_file.value())); } + else if (delete_file.value()->parsed_entry->isDeletionVector()) + { + deletion_vector_files.emplace_back(std::move(delete_file.value())); + } else { - position_deletes_files.emplace_back(std::move(delete_file.value())); + parquet_position_deletes_files.emplace_back(std::move(delete_file.value())); } delete_file = deletes_iterator.next(); } - LOG_DEBUG(logger, "Taken {} position deletes file and {} equality deletes files in iceberg iterator", position_deletes_files.size(), equality_deletes_files.size()); + LOG_DEBUG( + logger, + "Taken {} deletion vector files, {} parquet position delete files and {} equality delete files in iceberg iterator", + deletion_vector_files.size(), + parquet_position_deletes_files.size(), + equality_deletes_files.size()); std::sort(equality_deletes_files.begin(), equality_deletes_files.end()); - std::sort(position_deletes_files.begin(), position_deletes_files.end()); + std::sort(deletion_vector_files.begin(), deletion_vector_files.end()); + std::sort(parquet_position_deletes_files.begin(), parquet_position_deletes_files.end()); producer_task = std::make_unique( [this, thread_group = CurrentThread::getGroup()]() { @@ -334,42 +347,100 @@ ObjectInfoPtr IcebergIterator::next(size_t) manifest_file_entry, persistent_components.path_resolver.resolve(manifest_file_entry->parsed_entry->file_path_key), table_state_snapshot->schema_id); - for (const auto & position_delete : - defineDeletesSpan(manifest_file_entry, position_deletes_files, /* is_equality_delete */ false, logger)) + + const auto & data_file_path = object_info->info.data_object_file_path_key; + bool has_deletion_vector = false; + + for (const auto & deletion_vector : + defineDeletesSpan(manifest_file_entry, deletion_vector_files, /* is_equality_delete */ false, logger)) { - const auto & data_file_path = object_info->info.data_object_file_path_key; - const auto & lower = position_delete->parsed_entry->lower_reference_data_file_path; - const auto & upper = position_delete->parsed_entry->upper_reference_data_file_path; - bool can_contain_data_file_deletes - = (!lower.has_value() || *lower <= data_file_path) - && (!upper.has_value() || *upper >= data_file_path); - /// Skip position deletes that do not match the data file path. - if (!can_contain_data_file_deletes) + const auto & referenced_data_file = deletion_vector->parsed_entry->lower_reference_data_file_path; + if (!referenced_data_file.has_value() || referenced_data_file.value() != data_file_path) + continue; + + if (has_deletion_vector) + { + throw Exception( + ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION, + "Multiple deletion vectors match data file '{}'", + data_file_path); + } + + Iceberg::requireParquetDataFileForRowDeletes(object_info->info.file_format, "Deletion vectors"); + + if (!object_info->info.record_count.has_value()) { - ProfileEvents::increment(ProfileEvents::IcebergMinMaxPrunedDeleteFiles); - LOG_TEST( - logger, - "Skipping position delete file `{}` for data file `{}` because position delete has out of bounds reference data file " - "bounds: " - "(lower bound: `{}`, upper bound: `{}`)", - position_delete->parsed_entry->file_path_key, - data_file_path, - lower.has_value() ? lower->serialize() : "[no lower bound]", - upper.has_value() ? upper->serialize() : "[no upper bound]"); + throw Exception( + ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION, + "Data file '{}' is missing record_count required to validate deletion vector positions", + data_file_path); } - else + + const auto & parsed_entry = deletion_vector->parsed_entry; + /// For icebergCluster, next() runs on the initiator's task-distribution path: DV + /// I/O / CRC / roaring materialization happen here, then excluded_rows is sent on + /// the wire per task. Workers apply the bitmap and do not re-read the Puffin blob. + auto excluded_rows = Iceberg::loadDeletionVector( + object_storage, + persistent_components.path_resolver.resolve(parsed_entry->file_path_key), + parsed_entry->content_offset.value(), + parsed_entry->content_size_in_bytes.value(), + data_file_path, + referenced_data_file, + parsed_entry->record_count, + *object_info->info.record_count, + local_context, + logger); + + object_info->data_lake_metadata.emplace(); + if (excluded_rows) + object_info->data_lake_metadata->excluded_rows = std::move(excluded_rows); + has_deletion_vector = true; + LOG_DEBUG( + logger, + "Attached deletion vector from puffin file `{}` to data file `{}`", + parsed_entry->file_path_key, + data_file_path); + } + + if (!has_deletion_vector) + { + for (const auto & position_delete : + defineDeletesSpan(manifest_file_entry, parquet_position_deletes_files, /* is_equality_delete */ false, logger)) { - ProfileEvents::increment(ProfileEvents::IcebergMinMaxNonPrunedDeleteFiles); - LOG_TEST( - logger, - "Processing position delete file `{}` for data file `{}` with reference data file bounds: " - "(lower bound: `{}`, upper bound: `{}`)", - position_delete->parsed_entry->file_path_key, - data_file_path, - lower.has_value() ? lower->serialize() : "[no lower bound]", - upper.has_value() ? upper->serialize() : "[no upper bound]"); - object_info->addPositionDeleteObject( - position_delete, persistent_components.path_resolver.resolve(position_delete->parsed_entry->file_path_key)); + const auto & lower = position_delete->parsed_entry->lower_reference_data_file_path; + const auto & upper = position_delete->parsed_entry->upper_reference_data_file_path; + bool can_contain_data_file_deletes + = (!lower.has_value() || *lower <= data_file_path) + && (!upper.has_value() || *upper >= data_file_path); + /// Skip position deletes that do not match the data file path. + if (!can_contain_data_file_deletes) + { + ProfileEvents::increment(ProfileEvents::IcebergMinMaxPrunedDeleteFiles); + LOG_TEST( + logger, + "Skipping position delete file `{}` for data file `{}` because position delete has out of bounds reference data file " + "bounds: " + "(lower bound: `{}`, upper bound: `{}`)", + position_delete->parsed_entry->file_path_key, + data_file_path, + lower.has_value() ? lower->serialize() : "[no lower bound]", + upper.has_value() ? upper->serialize() : "[no upper bound]"); + } + else + { + ProfileEvents::increment(ProfileEvents::IcebergMinMaxNonPrunedDeleteFiles); + LOG_TEST( + logger, + "Processing position delete file `{}` for data file `{}` with reference data file bounds: " + "(lower bound: `{}`, upper bound: `{}`)", + position_delete->parsed_entry->file_path_key, + data_file_path, + lower.has_value() ? lower->serialize() : "[no lower bound]", + upper.has_value() ? upper->serialize() : "[no upper bound]"); + object_info->addPositionDeleteObject( + position_delete, persistent_components.path_resolver.resolve(position_delete->parsed_entry->file_path_key)); + } } } diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.h index a9d22bb54b12..9d53ed1e451a 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.h @@ -93,8 +93,10 @@ class IcebergIterator : public IObjectIterator ConcurrentBoundedQueue blocking_queue; std::unique_ptr producer_task; IDataLakeMetadata::FileProgressCallback callback; - std::vector position_deletes_files; + std::vector deletion_vector_files; + std::vector parquet_position_deletes_files; std::vector equality_deletes_files; + ContextPtr local_context; std::exception_ptr exception; std::mutex exception_mutex; Int32 table_schema_id; diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp index bd5686db3436..c03a2e14e582 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp @@ -10,6 +10,7 @@ #include #include #include +#include #include #include #include @@ -455,6 +456,7 @@ IcebergDataSnapshotPtr IcebergMetadata::createIcebergDataSnapshotFromSnapshotJSO std::optional total_rows; std::optional total_bytes; std::optional total_position_deletes; + std::optional total_equality_deletes; if (snapshot_object->has(f_summary)) { @@ -469,6 +471,9 @@ IcebergDataSnapshotPtr IcebergMetadata::createIcebergDataSnapshotFromSnapshotJSO { total_position_deletes = summary_object->getValue(f_total_position_deletes); } + + if (summary_object->has(f_total_equality_deletes)) + total_equality_deletes = summary_object->getValue(f_total_equality_deletes); } if (!snapshot_object->has(f_schema_id)) @@ -482,7 +487,8 @@ IcebergDataSnapshotPtr IcebergMetadata::createIcebergDataSnapshotFromSnapshotJSO schema_id, total_rows, total_bytes, - total_position_deletes); + total_position_deletes, + total_equality_deletes); } IcebergDataSnapshotPtr @@ -1195,30 +1201,67 @@ std::optional IcebergMetadata::totalRows(ContextPtr local_context) const return 0; } + /// Equality deletes remove data rows by value match; summary `total-equality-deletes` counts + /// rows in delete files, not deleted data rows. Fail closed when the field is present and > 0. + /// If the field is absent, skip the summary shortcut and scan manifests for EQUALITY_DELETE files. + if (actual_data_snapshot->total_equality_delete_rows.has_value() + && *actual_data_snapshot->total_equality_delete_rows > 0) + return {}; - /// All these "hints" with total rows or bytes are optional both in - /// metadata files and in manifest files, so we try all of them one by one - if (auto total_rows = actual_data_snapshot->getTotalRows(); total_rows.has_value()) + /// Prefer the snapshot-summary shortcut when equality deletes are explicitly zero. This avoids + /// opening every manifest for typical append-only tables (same fast path as before this PR). + if (actual_data_snapshot->allowsSnapshotTotalRowsShortcut()) { - ProfileEvents::increment(ProfileEvents::IcebergTrivialCountOptimizationApplied); - return total_rows; + if (auto total_rows = actual_data_snapshot->getTotalRows(); total_rows.has_value()) + { + ProfileEvents::increment(ProfileEvents::IcebergTrivialCountOptimizationApplied); + return total_rows; + } } - Int64 result = 0; + /// Fall through when the summary shortcut is unavailable. Manifest-list + /// `added_rows_count`/`existing_rows_count` are not used (some writers stamp them from + /// snapshot summary and can report 0 after compaction). Subtracting live position-delete / + /// deletion-vector `record_count` from data-file totals is also unsafe (duplicates, stale + /// references, DV supersession of parquet position deletes). Sum required per-data-file + /// `record_count` over live data files only when no live delete files exist; otherwise fail + /// closed to a real scan. + UInt64 result = 0; for (const auto & manifest_list_entry : actual_data_snapshot->manifest_list_entries) { auto manifest_file_ptr = getManifestFileEntriesHandle( object_storage, persistent_components, local_context, log, manifest_list_entry, actual_table_state_snapshot.schema_id); - auto data_count = manifest_file_ptr.getRowsCountInAllFilesExcludingDeleted(FileContentType::DATA); - auto position_deletes_count = manifest_file_ptr.getRowsCountInAllFilesExcludingDeleted(FileContentType::POSITION_DELETE); - if (!data_count.has_value() || !position_deletes_count.has_value()) + + if (!manifest_file_ptr.getFilesWithoutDeleted(FileContentType::EQUALITY_DELETE).empty() + || !manifest_file_ptr.getFilesWithoutDeleted(FileContentType::POSITION_DELETE).empty()) return {}; - result += data_count.value() - position_deletes_count.value(); + /// nullopt means a negative / overflowing per-file `record_count`: fail closed to a + /// real scan instead of returning a wrong count. Do not use optional column + /// `value_counts` here — nested fields can report element counts larger than rows. + auto manifest_rows = manifest_file_ptr.getRowsCountInAllFilesExcludingDeleted(FileContentType::DATA); + if (!manifest_rows.has_value()) + return {}; + /// Per-manifest sums are capped at Int64::max; still guard the cross-manifest total. + if (common::addOverflow(result, static_cast(*manifest_rows), result)) + return {}; + } + + if (auto summary_total_rows = actual_data_snapshot->getTotalRows(); + summary_total_rows.has_value() && *summary_total_rows != result) + { + LOG_WARNING( + log, + "Iceberg snapshot summary of table {} claims {} total rows, but its manifest files describe {} rows. " + "The snapshot summary is inconsistent with the table data (possibly a corrupted commit in the table " + "history), using the row count from the manifest files", + persistent_components.table_location, + *summary_total_rows, + result); } ProfileEvents::increment(ProfileEvents::IcebergTrivialCountOptimizationApplied); - return result; + return static_cast(result); } std::optional IcebergMetadata::totalBytes(ContextPtr local_context) const diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.cpp index f289f3d80d40..43fa5e26c3df 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.cpp @@ -3,6 +3,7 @@ #if USE_AVRO #include +#include #include @@ -13,6 +14,7 @@ namespace DB::ErrorCodes { extern const int LOGICAL_ERROR; + extern const int ICEBERG_SPECIFICATION_VIOLATION; } namespace DB::Iceberg @@ -32,6 +34,40 @@ String FileContentTypeToString(FileContentType type) throw DB::Exception(DB::ErrorCodes::LOGICAL_ERROR, "Unsupported content type: {}", static_cast(type)); } +std::optional getRecordCountInAllFilesExcludingDeleted( + const std::vector & files) +{ + Int64 result = 0; + for (const auto & file : files) + { + const Int64 record_count = file->parsed_entry->record_count; + if (record_count < 0) + return std::nullopt; + + const UInt64 record_count_u = static_cast(record_count); + const UInt64 result_u = static_cast(result); + if (result_u > static_cast(std::numeric_limits::max()) - record_count_u) + return std::nullopt; + + result += record_count; + } + return result; +} + +void requireDirectReferencedDataFileForPuffinDeletionVector( + bool set_from_referenced_data_file_field, + const std::optional & referenced_path, + const IcebergPathFromMetadata & manifest_file_path) +{ + if (!set_from_referenced_data_file_field || !referenced_path.has_value() || referenced_path->empty()) + { + throw DB::Exception( + DB::ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION, + "Puffin deletion vector entry in manifest file '{}' is missing referenced_data_file", + manifest_file_path); + } +} + static std::strong_ordering operator<=>(const PartitionSpecsEntry & lhs, const PartitionSpecsEntry & rhs) { return std::tie(lhs.source_id, lhs.transform_name, lhs.partition_name) diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.h index 1ce837d1e7f5..4b1fd9190f6d 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.h @@ -30,6 +30,7 @@ struct ColumnInfo #include #include +#include namespace DB::Iceberg { @@ -102,6 +103,17 @@ struct ParsedManifestFileEntry : boost::noncopyable Int64 record_count; Int64 file_size_in_bytes; + /// Iceberg v3 deletion vector metadata (position delete entries with puffin format) + std::optional content_offset; + std::optional content_size_in_bytes; + + bool isDeletionVector() const + { + return Poco::toLower(file_format) == "puffin" + && content_offset.has_value() + && content_size_in_bytes.has_value(); + } + ParsedManifestFileEntry( FileContentType content_type_, IcebergPathFromMetadata file_path_key_, @@ -118,7 +130,9 @@ struct ParsedManifestFileEntry : boost::noncopyable std::optional> equality_ids_, std::optional sort_order_id_, Int64 record_count_, - Int64 file_size_in_bytes_) + Int64 file_size_in_bytes_, + std::optional content_offset_ = std::nullopt, + std::optional content_size_in_bytes_ = std::nullopt) : content_type(content_type_) , file_path_key(std::move(file_path_key_)) , row_number(row_number_) @@ -135,6 +149,8 @@ struct ParsedManifestFileEntry : boost::noncopyable , sort_order_id(sort_order_id_) , record_count(record_count_) , file_size_in_bytes(file_size_in_bytes_) + , content_offset(content_offset_) + , content_size_in_bytes(content_size_in_bytes_) { } }; @@ -154,6 +170,19 @@ struct ProcessedManifestFileEntry using ProcessedManifestFileEntryPtr = std::shared_ptr; +/// Sum required per-file `record_count` over live manifest entries. +/// Returns nullopt if any entry has a negative `record_count` or the sum would overflow `Int64` +/// (fail closed — do not use optional column `value_counts`, which can disagree for nested fields). +std::optional getRecordCountInAllFilesExcludingDeleted( + const std::vector & files); + +/// Puffin deletion vectors must identify the data file via the dedicated `referenced_data_file` +/// manifest field (non-empty). Position-delete lower/upper bounds must not be used as a fallback. +void requireDirectReferencedDataFileForPuffinDeletionVector( + bool set_from_referenced_data_file_field, + const std::optional & referenced_path, + const IcebergPathFromMetadata & manifest_file_path); + bool operator<(const PartitionSpecification & lhs, const PartitionSpecification & rhs); bool operator<(const DB::Row & lhs, const DB::Row & rhs); diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFileIterator.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFileIterator.cpp index 0719ced8b60d..a4f905898d4d 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFileIterator.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFileIterator.cpp @@ -85,24 +85,7 @@ bool ManifestFileIterator::ManifestFileEntriesHandle::areAllDataFilesSortedBySor std::optional ManifestFileIterator::ManifestFileEntriesHandle::getRowsCountInAllFilesExcludingDeleted(FileContentType content) const { - Int64 result = 0; - for (const auto & file : getFilesWithoutDeleted(content)) - { - /// Have at least one column with rows count - bool found = false; - for (const auto & [column, column_info] : file->parsed_entry->columns_infos) - { - if (column_info.rows_count.has_value()) - { - result += *column_info.rows_count; - found = true; - break; - } - } - if (!found) - return std::nullopt; - } - return result; + return getRecordCountInAllFilesExcludingDeleted(getFilesWithoutDeleted(content)); } std::optional ManifestFileIterator::ManifestFileEntriesHandle::getBytesCountInAllDataFilesExcludingDeleted() const @@ -532,6 +515,11 @@ bool ManifestFileIterator::areAllDataFilesSortedBySortOrderID(Int32 sort_order_i return true; } +std::optional ManifestFileIterator::getRowsCountInAllFilesExcludingDeleted(FileContentType content) const +{ + return getFilesWithoutDeletedHandle().getRowsCountInAllFilesExcludingDeleted(content); +} + std::optional ManifestFileIterator::getBytesCountInAllDataFilesExcludingDeleted() const { Int64 result = 0; diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFileIterator.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFileIterator.h index ac98cc89de95..e109cd107890 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFileIterator.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFileIterator.h @@ -90,8 +90,8 @@ class ManifestFileIterator : public boost::noncopyable bool hasPartitionKey() const; const DB::KeyDescription & getPartitionKeyDescription() const; - /// Fields with rows count in manifest files are optional - /// they can be absent. + /// Sums required per-file `record_count` for live entries of the given content type. + /// Returns nullopt on negative `record_count` or Int64 overflow (fail closed). std::optional getRowsCountInAllFilesExcludingDeleted(FileContentType content) const; std::optional getBytesCountInAllDataFilesExcludingDeleted() const; diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/Mutations.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/Mutations.cpp index 8fb45abb86aa..c857ef6ca610 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/Mutations.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/Mutations.cpp @@ -41,6 +41,7 @@ namespace DB::ErrorCodes extern const int BAD_ARGUMENTS; extern const int LOGICAL_ERROR; extern const int LIMIT_EXCEEDED; +extern const int SUPPORT_IS_DISABLED; } namespace DB::DataLakeStorageSetting @@ -598,8 +599,20 @@ void mutate( auto metadata = getMetadataJSONObject(metadata_path, object_storage, persistent_table_components.metadata_cache, context, log, compression_method, persistent_table_components.table_uuid); - if (metadata->getValue(f_format_version) < 2) - throw Exception(ErrorCodes::BAD_ARGUMENTS, "Mutations are supported only for the second version of iceberg format"); + /// Iceberg v3 writers must not add new position-delete files; row-level deletes require + /// deletion vectors. Fail closed before any object writes until ClickHouse can write DVs. + const Int32 format_version = metadata->getValue(f_format_version); + if (format_version < 2) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Mutations are supported only for Iceberg format version 2"); + if (format_version >= 3) + { + throw Exception( + ErrorCodes::SUPPORT_IS_DISABLED, + "Iceberg DELETE and UPDATE mutations are not supported for format version {}. " + "ClickHouse writes parquet position-delete files, which Iceberg v3+ writers must not add; " + "writing deletion vectors is not implemented yet", + format_version); + } auto partition_spec_id = metadata->getValue(Iceberg::f_default_spec_id); auto partitions_specs = metadata->getArray(Iceberg::f_partition_specs); Poco::JSON::Object::Ptr partititon_spec; diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/Snapshot.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/Snapshot.h index fd46422f60ad..c8ee62b8ef36 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/Snapshot.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/Snapshot.h @@ -16,17 +16,34 @@ struct IcebergDataSnapshot DB::ManifestFileCacheKeys manifest_list_entries; Int64 snapshot_id; Int64 schema_id_on_snapshot_commit; + /// From snapshot summary (`total-records`). Preferred by the trivial COUNT shortcut when + /// `allowsSnapshotTotalRowsShortcut` holds; otherwise compared to the manifest-derived count + /// for a mismatch warning. Summary totals are maintained incrementally by writers and can be + /// poisoned by a bad commit in table history. std::optional total_rows; std::optional total_bytes; std::optional total_position_delete_rows; + /// Rows in equality-delete files (snapshot summary). Not a count of deleted data rows; + /// used only to fail closed / gate the trivial COUNT shortcut. + std::optional total_equality_delete_rows; std::optional partition_key; std::optional sorting_key; std::optional getTotalRows() const { - if (total_rows.has_value() && total_position_delete_rows.has_value()) - return *total_rows - *total_position_delete_rows; - return std::nullopt; + if (!total_rows.has_value() || !total_position_delete_rows.has_value()) + return std::nullopt; + /// Fail closed on inconsistent summary: unsigned subtract would wrap to a huge COUNT. + if (*total_position_delete_rows > *total_rows) + return std::nullopt; + return *total_rows - *total_position_delete_rows; + } + + /// Summary `total-equality-deletes` is optional. Only trust the cheap `getTotalRows` shortcut + /// when the field is present and explicitly zero; absent or >0 must fall through / fail closed. + bool allowsSnapshotTotalRowsShortcut() const + { + return total_equality_delete_rows.has_value() && *total_equality_delete_rows == 0; } }; diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_count_shortcuts.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_count_shortcuts.cpp new file mode 100644 index 000000000000..0850e5141754 --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_count_shortcuts.cpp @@ -0,0 +1,163 @@ +#include + +#include "config.h" + +#if USE_AVRO + +#include +#include +#include +#include + +#include + +using namespace DB; +using namespace DB::Iceberg; + +TEST(IcebergCountShortcuts, HasEqualityAndPositionDeleteHelpers) +{ + Iceberg::IcebergObjectSerializableInfo info; + info.data_object_file_path_key = Iceberg::IcebergPathFromMetadata::deserialize("s3://bucket/data/file.parquet"); + info.file_format = "PARQUET"; + + auto plain = std::make_shared(RelativePathWithMetadata{"data/file.parquet"}); + EXPECT_FALSE(hasIcebergEqualityDeletes(plain)); + EXPECT_FALSE(hasIcebergPositionDeletes(plain)); + + auto iceberg = std::make_shared(RelativePathWithMetadata{"data/file.parquet"}, info); + EXPECT_FALSE(hasIcebergEqualityDeletes(iceberg)); + EXPECT_FALSE(hasIcebergPositionDeletes(iceberg)); + + iceberg->info.equality_deletes_objects.push_back( + Iceberg::EqualityDeleteObject{ + .file_path = "s3://bucket/deletes/eq.parquet", + .file_format = "PARQUET", + .equality_ids = std::vector{1}, + .schema_id = 0, + }); + EXPECT_TRUE(hasIcebergEqualityDeletes(iceberg)); + EXPECT_FALSE(hasIcebergPositionDeletes(iceberg)); + + iceberg->info.position_deletes_objects.push_back( + Iceberg::PositionDeleteObject{ + .file_path = "s3://bucket/deletes/pos.parquet", + .file_format = "PARQUET", + .reference_data_file_path = std::nullopt, + .sequence_number = 1, + }); + EXPECT_TRUE(hasIcebergEqualityDeletes(iceberg)); + EXPECT_TRUE(hasIcebergPositionDeletes(iceberg)); +} + +TEST(IcebergCountShortcuts, SnapshotSummaryShortcutRequiresExplicitZeroEqualityDeletes) +{ + Iceberg::IcebergDataSnapshot snapshot; + snapshot.total_rows = 100; + snapshot.total_position_delete_rows = 10; + snapshot.total_equality_delete_rows = 0; + + EXPECT_TRUE(snapshot.allowsSnapshotTotalRowsShortcut()); + ASSERT_TRUE(snapshot.getTotalRows().has_value()); + EXPECT_EQ(*snapshot.getTotalRows(), 90u); + + snapshot.total_equality_delete_rows = std::nullopt; + EXPECT_FALSE(snapshot.allowsSnapshotTotalRowsShortcut()); + + snapshot.total_equality_delete_rows = 1; + EXPECT_FALSE(snapshot.allowsSnapshotTotalRowsShortcut()); +} + +TEST(IcebergCountShortcuts, GetTotalRowsFailsClosedWhenPositionDeletesExceedRows) +{ + Iceberg::IcebergDataSnapshot snapshot; + snapshot.total_rows = 5; + snapshot.total_position_delete_rows = 6; + EXPECT_FALSE(snapshot.getTotalRows().has_value()); +} + +namespace +{ + +ProcessedManifestFileEntryPtr makeDataEntryForRecordCount( + Int64 record_count, + std::unordered_map columns_infos = {}) +{ + auto parsed = std::make_shared( + FileContentType::DATA, + IcebergPathFromMetadata::deserialize("s3://bucket/data/file.parquet"), + /*row_number=*/0, + ManifestEntryStatus::ADDED, + /*written_sequence_number=*/std::nullopt, + /*written_snapshot_id=*/std::nullopt, + DB::Row{}, + std::move(columns_infos), + std::unordered_map>{}, + /*file_format=*/"PARQUET", + /*lower_reference_data_file_path=*/std::nullopt, + /*upper_reference_data_file_path=*/std::nullopt, + /*equality_ids=*/std::nullopt, + /*sort_order_id=*/std::nullopt, + record_count, + /*file_size_in_bytes=*/100); + + auto processed = std::make_shared(); + processed->parsed_entry = std::move(parsed); + processed->common_partition_specification = std::make_shared(); + processed->sequence_number = 0; + processed->resolved_schema_id = 0; + processed->manifest_file_path = "s3://bucket/metadata/manifest.avro"; + return processed; +} + +} + +TEST(IcebergRecordCountAggregate, SumsRecordCountIgnoringValueCounts) +{ + ColumnInfo nested_list_stats; + nested_list_stats.rows_count = 1000; /// nested element count, not row count + + const auto total = getRecordCountInAllFilesExcludingDeleted({ + makeDataEntryForRecordCount(/*record_count=*/10, {{/*column_id=*/2, nested_list_stats}}), + makeDataEntryForRecordCount(/*record_count=*/5), + }); + + ASSERT_TRUE(total.has_value()); + EXPECT_EQ(*total, 15); +} + +TEST(IcebergRecordCountAggregate, SucceedsWithoutValueCounts) +{ + const auto total = getRecordCountInAllFilesExcludingDeleted({ + makeDataEntryForRecordCount(/*record_count=*/42), + }); + + ASSERT_TRUE(total.has_value()); + EXPECT_EQ(*total, 42); +} + +TEST(IcebergRecordCountAggregate, EmptyManifestIsZero) +{ + const auto total = getRecordCountInAllFilesExcludingDeleted({}); + ASSERT_TRUE(total.has_value()); + EXPECT_EQ(*total, 0); +} + +TEST(IcebergRecordCountAggregate, NegativeRecordCountFailsClosed) +{ + const auto total = getRecordCountInAllFilesExcludingDeleted({ + makeDataEntryForRecordCount(/*record_count=*/10), + makeDataEntryForRecordCount(/*record_count=*/-1), + }); + EXPECT_FALSE(total.has_value()); +} + +TEST(IcebergRecordCountAggregate, OverflowFailsClosed) +{ + const auto total = getRecordCountInAllFilesExcludingDeleted({ + makeDataEntryForRecordCount(/*record_count=*/std::numeric_limits::max()), + makeDataEntryForRecordCount(/*record_count=*/1), + }); + EXPECT_FALSE(total.has_value()); +} + +#endif diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_data_object_info_clone.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_data_object_info_clone.cpp new file mode 100644 index 000000000000..b863ee605446 --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_data_object_info_clone.cpp @@ -0,0 +1,64 @@ +#include + +#include "config.h" + +#if USE_AVRO + +#include +#include +#include +#include + +using namespace DB; + +TEST(IcebergDataObjectInfoClone, PreservesEqualityAndPositionDeletes) +{ + Iceberg::IcebergObjectSerializableInfo info; + info.data_object_file_path_key = Iceberg::IcebergPathFromMetadata::deserialize("s3://bucket/data/file.parquet"); + info.file_format = "PARQUET"; + info.equality_deletes_objects.push_back( + Iceberg::EqualityDeleteObject{ + .file_path = "s3://bucket/deletes/eq.parquet", + .file_format = "PARQUET", + .equality_ids = std::vector{1, 2}, + .schema_id = 7, + }); + info.position_deletes_objects.push_back( + Iceberg::PositionDeleteObject{ + .file_path = "s3://bucket/deletes/pos.parquet", + .file_format = "PARQUET", + .reference_data_file_path = std::nullopt, + .sequence_number = 42, + }); + + auto original = std::make_shared( + RelativePathWithMetadata{"data/file.parquet"}, info); + original->data_lake_metadata.emplace(); + original->data_lake_metadata->excluded_rows = std::make_shared(); + original->data_lake_metadata->excluded_rows->add(11); + + /// Object-slicing (the old ObjectIteratorSplitByBuckets path) drops Iceberg metadata. + ObjectInfo sliced_value = *original; + auto sliced = std::make_shared(sliced_value); + EXPECT_FALSE(std::dynamic_pointer_cast(sliced)); + ASSERT_TRUE(sliced->data_lake_metadata.has_value()); + ASSERT_TRUE(sliced->data_lake_metadata->excluded_rows); + EXPECT_EQ(sliced->data_lake_metadata->excluded_rows->size(), 1u); + + auto cloned = original->clone(); + auto iceberg_cloned = std::dynamic_pointer_cast(cloned); + ASSERT_TRUE(iceberg_cloned); + ASSERT_EQ(iceberg_cloned->info.equality_deletes_objects.size(), 1u); + EXPECT_EQ(iceberg_cloned->info.equality_deletes_objects[0].file_path, "s3://bucket/deletes/eq.parquet"); + ASSERT_TRUE(iceberg_cloned->info.equality_deletes_objects[0].equality_ids.has_value()); + EXPECT_EQ(iceberg_cloned->info.equality_deletes_objects[0].equality_ids->size(), 2u); + ASSERT_EQ(iceberg_cloned->info.position_deletes_objects.size(), 1u); + EXPECT_EQ(iceberg_cloned->info.position_deletes_objects[0].file_path, "s3://bucket/deletes/pos.parquet"); + EXPECT_EQ(iceberg_cloned->info.position_deletes_objects[0].sequence_number, 42); + ASSERT_TRUE(iceberg_cloned->data_lake_metadata.has_value()); + ASSERT_TRUE(iceberg_cloned->data_lake_metadata->excluded_rows); + EXPECT_EQ(iceberg_cloned->data_lake_metadata->excluded_rows->size(), 1u); + EXPECT_TRUE(iceberg_cloned->data_lake_metadata->excluded_rows->rb_contains(11)); +} + +#endif diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_deletion_vector_positions.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_deletion_vector_positions.cpp new file mode 100644 index 000000000000..fa3442c4fad7 --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_deletion_vector_positions.cpp @@ -0,0 +1,90 @@ +#include + +#include "config.h" + +#if USE_AVRO + +#include +#include + +#include + +using namespace DB; +using namespace DB::Iceberg; + +namespace DB +{ +namespace ErrorCodes +{ +extern const int ICEBERG_SPECIFICATION_VIOLATION; +} +} + +TEST(IcebergDeletionVectorPositions, AcceptsBoundaryPosition) +{ + /// Valid file-local positions are in [0, record_count). + const std::vector positions = {0, 9}; + EXPECT_NO_THROW(validateDeletionVectorPositionsAgainstDataFile(positions, /*expected_cardinality=*/2, /*data_file_record_count=*/10)); +} + +TEST(IcebergDeletionVectorPositions, RejectsPositionEqualToRecordCount) +{ + const std::vector positions = {10}; + try + { + validateDeletionVectorPositionsAgainstDataFile(positions, /*expected_cardinality=*/1, /*data_file_record_count=*/10); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION); + EXPECT_NE(e.message().find("out of range"), std::string::npos); + } +} + +TEST(IcebergDeletionVectorPositions, RejectsPositionAboveRecordCount) +{ + const std::vector positions = {11}; + try + { + validateDeletionVectorPositionsAgainstDataFile(positions, /*expected_cardinality=*/1, /*data_file_record_count=*/10); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION); + EXPECT_NE(e.message().find("out of range"), std::string::npos); + } +} + +TEST(IcebergDeletionVectorPositions, RejectsCardinalityExceedingRecordCount) +{ + const std::vector positions; + try + { + validateDeletionVectorPositionsAgainstDataFile(positions, /*expected_cardinality=*/11, /*data_file_record_count=*/10); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION); + EXPECT_NE(e.message().find("exceeds data file record_count"), std::string::npos); + } +} + +TEST(IcebergDeletionVectorPositions, RejectsNegativeDataFileRecordCount) +{ + const std::vector positions; + try + { + validateDeletionVectorPositionsAgainstDataFile(positions, /*expected_cardinality=*/0, /*data_file_record_count=*/-1); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION); + EXPECT_NE(e.message().find("non-negative"), std::string::npos); + } +} + +#endif diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_parquet_row_deletes_guard.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_parquet_row_deletes_guard.cpp new file mode 100644 index 000000000000..ce5262728d1b --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_parquet_row_deletes_guard.cpp @@ -0,0 +1,47 @@ +#include + +#include +#include + +using namespace DB; + +namespace DB +{ +namespace ErrorCodes +{ +extern const int NOT_IMPLEMENTED; +} +} + +TEST(IcebergParquetRowDeletesGuard, AcceptsParquet) +{ + EXPECT_NO_THROW(Iceberg::requireParquetDataFileForRowDeletes("parquet", "Deletion vectors")); + EXPECT_NO_THROW(Iceberg::requireParquetDataFileForRowDeletes("PARQUET", "Position deletes")); + EXPECT_NO_THROW(Iceberg::requireParquetDataFileForRowDeletes("Parquet", "Deletion vectors")); +} + +TEST(IcebergParquetRowDeletesGuard, RejectsNonParquet) +{ + try + { + Iceberg::requireParquetDataFileForRowDeletes("ORC", "Deletion vectors"); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::NOT_IMPLEMENTED); + EXPECT_NE(e.message().find("Deletion vectors are only supported"), std::string::npos); + EXPECT_NE(e.message().find("ORC"), std::string::npos); + } + + try + { + Iceberg::requireParquetDataFileForRowDeletes("AVRO", "Position deletes"); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::NOT_IMPLEMENTED); + EXPECT_NE(e.message().find("Position deletes are only supported"), std::string::npos); + } +} diff --git a/src/Storages/ObjectStorage/DataLakes/PuffinDeletionVectorReader.cpp b/src/Storages/ObjectStorage/DataLakes/PuffinDeletionVectorReader.cpp new file mode 100644 index 000000000000..b91a87fc6269 --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/PuffinDeletionVectorReader.cpp @@ -0,0 +1,297 @@ +#include +#include + +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +namespace ProfileEvents +{ +extern const Event PuffinFilesRead; +extern const Event PuffinFileReadMicroseconds; +} + +namespace DB +{ + +namespace ErrorCodes +{ + extern const int BAD_ARGUMENTS; +} + +void validateDeletionVectorV1Fields(const std::vector & fields, size_t blob_index) +{ + if (fields.empty()) + return; + + /// Spark / Iceberg file-scoped DVs use the reserved `_pos` id as a singleton marker. + if (fields.size() == 1 && fields[0] == ICEBERG_ROW_POSITION_FIELD_ID) + return; + + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Puffin blob {}: deletion-vector-v1 has unsupported non-empty 'fields' " + "(only [] or [{}] / Iceberg _pos are accepted; column-scoped DVs are not supported)", + blob_index, + ICEBERG_ROW_POSITION_FIELD_ID); +} + +namespace +{ + +struct ScopedPuffinFileReadProfileEvent +{ + ProfileEventTimeIncrement watch; + + ScopedPuffinFileReadProfileEvent() + : watch(ProfileEvents::PuffinFileReadMicroseconds) + { + ProfileEvents::increment(ProfileEvents::PuffinFilesRead); + } +}; + +UInt32 readBigEndianUInt32(const UInt8 * data) +{ + return (static_cast(data[0]) << 24) + | (static_cast(data[1]) << 16) + | (static_cast(data[2]) << 8) + | static_cast(data[3]); +} + +} + +void validatePuffinBlobBounds(Int64 offset, Int64 length, size_t file_size, std::string_view context) +{ + if (offset < 0 || length < 0) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "{}: offset/length out of bounds", context); + + if (offset > static_cast(file_size) || length > static_cast(file_size)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "{}: offset/length out of bounds", context); + + Int64 end = 0; + if (common::addOverflow(offset, length, end)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "{}: offset/length out of bounds", context); + + if (static_cast(end) > file_size) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "{}: offset/length out of bounds", context); +} + +void checkDeletionVectorBlobReadLimits(Int64 length, std::optional expected_cardinality) +{ + /// Same fail-closed order as the SQL `Puffin` path: cardinality before blob length / allocate. + if (expected_cardinality.has_value() && *expected_cardinality > PUFFIN_DV_MAX_MATERIALIZED_POSITIONS) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Deletion vector cardinality {} exceeds materialization limit {}", + *expected_cardinality, + PUFFIN_DV_MAX_MATERIALIZED_POSITIONS); + + if (length < 0) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector blob length is negative"); + + if (static_cast(length) > PUFFIN_DV_MAX_BLOB_SIZE) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Deletion vector blob length {} exceeds absolute limit {}", + length, + PUFFIN_DV_MAX_BLOB_SIZE); + + if (static_cast(length) < 12) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector blob is too small"); +} + +void validateDeletionVectorEnvelope(const UInt8 * header, Int64 length) +{ + const UInt32 combined_length = readBigEndianUInt32(header); + if (std::memcmp(header + sizeof(UInt32), DELETION_VECTOR_MAGIC, sizeof(DELETION_VECTOR_MAGIC)) != 0) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid deletion vector magic"); + + if (combined_length < sizeof(DELETION_VECTOR_MAGIC)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid deletion vector combined length: {}", combined_length); + + UInt64 expected_blob_size = 0; + if (common::addOverflow(static_cast(combined_length), UInt64{8}, expected_blob_size)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid deletion vector combined length: {}", combined_length); + + if (static_cast(length) != expected_blob_size) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Deletion vector blob size {} does not match combined length {}", + length, + combined_length); +} + +std::vector deserializeDeletionVectorV1Blob(std::string_view blob_bytes, std::optional expected_cardinality) +{ + if (expected_cardinality.has_value() && *expected_cardinality > PUFFIN_DV_MAX_MATERIALIZED_POSITIONS) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Deletion vector cardinality {} exceeds materialization limit {}", + *expected_cardinality, + PUFFIN_DV_MAX_MATERIALIZED_POSITIONS); + + return deserializeRoaringPositionBitmap(extractDeletionVectorPayload(blob_bytes), expected_cardinality); +} + +std::vector readDeletionVectorFromPuffin(ReadBuffer & file, Int64 offset, Int64 length, std::optional expected_cardinality) +{ + ScopedPuffinFileReadProfileEvent profile_event; + + checkDeletionVectorBlobReadLimits(length, expected_cardinality); + + if (auto file_size = tryGetFileSizeFromReadBuffer(file)) + validatePuffinBlobBounds(offset, length, *file_size); + else if (offset < 0) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid Puffin deletion vector offset {} or length {}", offset, length); + + auto * seekable = dynamic_cast(&file); + if (!seekable) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Puffin deletion vector read requires a seekable buffer"); + + /// Peek combined_length + magic before allocating `length` (up to 2 GiB). Matches + /// `readDeletionVectorBlobBytes` in the SQL Puffin format path. + seekable->seek(offset, SEEK_SET); + + UInt8 header[8]; + file.readStrict(reinterpret_cast(header), sizeof(header)); + validateDeletionVectorEnvelope(header, length); + + String blob_data(static_cast(length), '\0'); + std::memcpy(blob_data.data(), header, sizeof(header)); + file.readStrict(blob_data.data() + sizeof(header), blob_data.size() - sizeof(header)); + + return deserializeDeletionVectorV1Blob(blob_data, expected_cardinality); +} + +void appendReadBufferWithAbsoluteSizeLimit(ReadBuffer & buf, std::vector & out, size_t max_buffered_size) +{ + if (out.size() > max_buffered_size) + { + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Puffin non-seekable buffer size {} exceeds absolute limit {}", + out.size(), + max_buffered_size); + } + + std::vector tmp(DBMS_DEFAULT_BUFFER_SIZE); + while (!buf.eof()) + { + const size_t capacity = max_buffered_size - out.size(); + if (capacity == 0) + { + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Puffin non-seekable input exceeds absolute buffer limit {} bytes; use seekable input for larger files", + max_buffered_size); + } + + const size_t to_read = std::min(tmp.size(), capacity); + const size_t n = buf.read(reinterpret_cast(tmp.data()), to_read); + if (n == 0) + break; + + out.insert(out.end(), tmp.data(), tmp.data() + n); + + /// If we filled the remaining capacity and the stream still has data, fail closed. + if (n == capacity && !buf.eof()) + { + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Puffin non-seekable input exceeds absolute buffer limit {} bytes; use seekable input for larger files", + max_buffered_size); + } + } +} + +const PuffinBlob & bindDeletionVectorBlob( + const std::vector & blobs, + Int64 content_offset, + Int64 content_size_in_bytes, + std::string_view expected_referenced_data_file, + UInt64 expected_cardinality) +{ + const PuffinBlob * matched = nullptr; + size_t matched_index = 0; + + for (size_t i = 0; i < blobs.size(); ++i) + { + if (blobs[i].offset != content_offset || blobs[i].length != content_size_in_bytes) + continue; + + if (matched) + { + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Multiple Puffin blobs claim offset {} length {}", + content_offset, + content_size_in_bytes); + } + + matched = &blobs[i]; + matched_index = i; + } + + if (!matched) + { + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "No Puffin footer blob at offset {} length {}", + content_offset, + content_size_in_bytes); + } + + if (matched->type != "deletion-vector-v1") + { + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Puffin blob {} at offset {} length {} has type '{}', expected deletion-vector-v1", + matched_index, + content_offset, + content_size_in_bytes, + matched->type); + } + + if (!matched->compression_codec.empty()) + { + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Puffin blob {}: deletion-vector-v1 must omit compression-codec", + matched_index); + } + + /// Structural DV footer checks live in the pre-existing SQL `Puffin` helper. + const UInt64 footer_cardinality = requireDeletionVectorV1Properties(*matched, matched_index); + + const auto & referenced_data_file = matched->properties.at("referenced-data-file"); + if (referenced_data_file != expected_referenced_data_file) + { + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Puffin blob {} referenced-data-file '{}' does not match expected data file '{}'", + matched_index, + referenced_data_file, + expected_referenced_data_file); + } + + if (footer_cardinality != expected_cardinality) + { + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Puffin blob {} cardinality {} does not match expected cardinality {}", + matched_index, + footer_cardinality, + expected_cardinality); + } + + return *matched; +} + +} diff --git a/src/Storages/ObjectStorage/DataLakes/PuffinDeletionVectorReader.h b/src/Storages/ObjectStorage/DataLakes/PuffinDeletionVectorReader.h new file mode 100644 index 000000000000..b78f57041d02 --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/PuffinDeletionVectorReader.h @@ -0,0 +1,80 @@ +#pragma once + +#include +#include +#include + +#include +#include +#include +#include + +namespace DB +{ + +/// Absolute cap on on-disk deletion-vector-v1 blob length before reading bytes into memory. +/// Aligns with Iceberg DeleteLoader's 2 GiB content-size check. +constexpr size_t PUFFIN_DV_MAX_BLOB_SIZE = 2ULL * 1024 * 1024 * 1024; +/// Absolute cap on materialized deleted positions (~800 MiB of UInt64s at this limit). +constexpr UInt64 PUFFIN_DV_MAX_MATERIALIZED_POSITIONS = 100'000'000; + +/// Leading / footer-open magic is always 4 bytes (`PFA1`). +constexpr size_t PUFFIN_MAGIC_SIZE = 4; +constexpr size_t PUFFIN_FOOTER_TRAILER_SIZE = 12; +/// Absolute cap on footer payload size (uncompressed JSON bytes, or declared LZ4 contentSize). +constexpr size_t PUFFIN_FOOTER_MAX_PAYLOAD_SIZE = 16 * 1024 * 1024; + +/// Non-seekable SQL path must buffer the whole file to reach the trailer. Cap total buffered size +/// so a crafted pipe cannot allocate unbounded memory before footer-length validation. +/// Sized for one max DV blob + max footer (header magic + blob + footer magic + payload + trailer). +constexpr size_t PUFFIN_NON_SEEKABLE_MAX_BUFFERED_SIZE = PUFFIN_MAGIC_SIZE + PUFFIN_DV_MAX_BLOB_SIZE + + PUFFIN_MAGIC_SIZE + PUFFIN_FOOTER_MAX_PAYLOAD_SIZE + PUFFIN_FOOTER_TRAILER_SIZE; + +/// Iceberg reserved `_pos` field id (`std::numeric_limits::max() - 2`). Spark / Iceberg +/// writers put this singleton in deletion-vector-v1 puffin `fields` for file-scoped DVs. +constexpr Int32 ICEBERG_ROW_POSITION_FIELD_ID = std::numeric_limits::max() - 2; + +/// File-scoped DVs may use `fields=[]` or `fields=[ICEBERG_ROW_POSITION_FIELD_ID]`. +/// Any other list is treated as unsupported column-scoped deletion vectors. +void validateDeletionVectorV1Fields(const std::vector & fields, size_t blob_index); + +/// Validate that [offset, offset + length) fits within file_size. +void validatePuffinBlobBounds(Int64 offset, Int64 length, size_t file_size, std::string_view context = "Puffin deletion vector"); + +/// Iceberg deletion-vector-v1 envelope magic (`0xD1D33964`), shared with SQL `Puffin` decode. +inline constexpr UInt8 DELETION_VECTOR_MAGIC[4] = {0xD1, 0xD3, 0x39, 0x64}; + +/// Fail closed before envelope peek / full allocate. Shared by SQL `Puffin` and Iceberg loaders. +/// Order: cardinality ceiling, then length bounds (`length < 0`, absolute blob cap, min envelope). +void checkDeletionVectorBlobReadLimits(Int64 length, std::optional expected_cardinality); + +/// Validate deletion-vector-v1 envelope (combined_length + magic) against declared blob `length`. +/// `header` must point at the first 8 bytes of the blob. Throws on mismatch before a full allocate. +void validateDeletionVectorEnvelope(const UInt8 * header, Int64 length); + +/// Deserialize a deletion-vector-v1 blob (magic + CRC wrapper + roaring bitmap payload). +std::vector deserializeDeletionVectorV1Blob(std::string_view blob_bytes, std::optional expected_cardinality = std::nullopt); + +/// Read a deletion-vector-v1 blob from a Puffin file at the given offset and length. +std::vector readDeletionVectorFromPuffin(ReadBuffer & file, Int64 offset, Int64 length, std::optional expected_cardinality = std::nullopt); + +/// Append bytes from `buf` into `out` until EOF. Throws if `out` would exceed `max_buffered_size`. +void appendReadBufferWithAbsoluteSizeLimit(ReadBuffer & buf, std::vector & out, size_t max_buffered_size); + +/// Compatibility alias for Iceberg / tests: same as `readPuffinFooterFromSeekable`. +inline std::vector readPuffinFooterBlobsFromSeekable(SeekableReadBuffer & seekable, size_t file_size) +{ + return readPuffinFooterFromSeekable(seekable, file_size); +} + +/// Find the unique footer blob at (`content_offset`, `content_size_in_bytes`) and bind it as a +/// deletion-vector-v1 for `expected_referenced_data_file` with `expected_cardinality`. +/// Throws if the slice is missing, ambiguous, or does not match the expected DV identity. +const PuffinBlob & bindDeletionVectorBlob( + const std::vector & blobs, + Int64 content_offset, + Int64 content_size_in_bytes, + std::string_view expected_referenced_data_file, + UInt64 expected_cardinality); + +} diff --git a/src/Storages/ObjectStorage/DataLakes/PuffinFilesCache.cpp b/src/Storages/ObjectStorage/DataLakes/PuffinFilesCache.cpp new file mode 100644 index 000000000000..5cb9038c0b44 --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/PuffinFilesCache.cpp @@ -0,0 +1,276 @@ +#include + +#include +#include +#include +#include + +#include +#include + +namespace CurrentMetrics +{ +extern const Metric PuffinFilesCacheBytes; +extern const Metric PuffinFilesCacheFiles; +} + +namespace ProfileEvents +{ +extern const Event PuffinFilesCacheWeightLost; +} + +namespace DB +{ + +namespace +{ + +constexpr size_t FOOTER_MEMO_ENTRY_OVERHEAD = 256; +constexpr size_t FOOTER_BLOB_OVERHEAD = 64; + +UInt64 saturatingAdd(UInt64 left, UInt64 right) +{ + UInt64 result = 0; + if (common::addOverflow(left, right, result)) + return std::numeric_limits::max(); + return result; +} + +UInt64 saturatingAdd(std::initializer_list values) +{ + UInt64 result = 0; + for (UInt64 value : values) + result = saturatingAdd(result, value); + return result; +} + +} + +DataLakeObjectMetadata::ExcludedRowsPtr PuffinFilesCache::cloneExcludedRows(const PuffinFilesCacheCell & cell) +{ + if (cell.is_empty_deletion_vector) + return nullptr; + + auto cloned = std::make_shared(); + cloned->merge(*cell.excluded_rows); + return cloned; +} + +bool PuffinFilesCacheKey::operator==(const PuffinFilesCacheKey & other) const +{ + return storage_identity == other.storage_identity + && file_path == other.file_path + && etag == other.etag + && content_offset == other.content_offset + && content_size_in_bytes == other.content_size_in_bytes + && referenced_data_file == other.referenced_data_file + && expected_cardinality == other.expected_cardinality + && data_file_record_count == other.data_file_record_count; +} + +UInt64 PuffinFilesCacheKey::approximateMemoryBytes() const +{ + /// Charge string payloads plus the key object / hash-map slot baseline. + return saturatingAdd( + {static_cast(sizeof(PuffinFilesCacheKey)), + static_cast(storage_identity.size()), + static_cast(file_path.size()), + static_cast(etag.size()), + static_cast(referenced_data_file.size())}); +} + +size_t PuffinFilesCacheKeyHash::operator()(const PuffinFilesCacheKey & key) const +{ + size_t hash = 0; + boost::hash_combine(hash, CityHash_v1_0_2::CityHash64(key.storage_identity.data(), key.storage_identity.size())); + boost::hash_combine(hash, CityHash_v1_0_2::CityHash64(key.file_path.data(), key.file_path.size())); + boost::hash_combine(hash, CityHash_v1_0_2::CityHash64(key.etag.data(), key.etag.size())); + boost::hash_combine(hash, key.content_offset); + boost::hash_combine(hash, key.content_size_in_bytes); + boost::hash_combine(hash, CityHash_v1_0_2::CityHash64(key.referenced_data_file.data(), key.referenced_data_file.size())); + boost::hash_combine(hash, key.expected_cardinality); + boost::hash_combine(hash, key.data_file_record_count); + return hash; +} + +UInt64 PuffinFilesCacheCell::calculateMemorySize( + bool is_empty_deletion_vector_, + const DataLakeObjectMetadata::ExcludedRowsPtr & excluded_rows_, + UInt64 key_memory_bytes_) +{ + const UInt64 payload_bytes = is_empty_deletion_vector_ + ? 0 + : (excluded_rows_ ? excluded_rows_->getAllocatedBytes() : 0); + + return saturatingAdd( + {key_memory_bytes_, + payload_bytes, + static_cast(sizeof(PuffinFilesCacheCell)), + static_cast(SIZE_IN_MEMORY_OVERHEAD)}); +} + +PuffinFilesCacheCell::PuffinFilesCacheCell(DataLakeObjectMetadata::ExcludedRowsPtr excluded_rows_, UInt64 key_memory_bytes_) + : excluded_rows(std::move(excluded_rows_)) + , is_empty_deletion_vector(!excluded_rows) + , memory_bytes(calculateMemorySize(is_empty_deletion_vector, excluded_rows, key_memory_bytes_)) +{ +} + +size_t PuffinFilesCacheWeightFunction::operator()(const PuffinFilesCacheCell & cell) const +{ + return cell.memory_bytes; +} + +bool PuffinFooterCacheKey::operator==(const PuffinFooterCacheKey & other) const +{ + return storage_identity == other.storage_identity && file_path == other.file_path && etag == other.etag; +} + +size_t PuffinFooterCacheKeyHash::operator()(const PuffinFooterCacheKey & key) const +{ + size_t hash = 0; + boost::hash_combine(hash, CityHash_v1_0_2::CityHash64(key.storage_identity.data(), key.storage_identity.size())); + boost::hash_combine(hash, CityHash_v1_0_2::CityHash64(key.file_path.data(), key.file_path.size())); + boost::hash_combine(hash, CityHash_v1_0_2::CityHash64(key.etag.data(), key.etag.size())); + return hash; +} + +UInt64 PuffinFilesCache::approximateFooterEntryBytes(const PuffinFooterCacheKey & key, const FooterBlobsPtr & blobs) +{ + UInt64 bytes = saturatingAdd( + {static_cast(sizeof(PuffinFooterCacheKey)), + static_cast(key.storage_identity.size()), + static_cast(key.file_path.size()), + static_cast(key.etag.size()), + static_cast(sizeof(FooterMemoEntry)), + static_cast(FOOTER_MEMO_ENTRY_OVERHEAD)}); + + if (!blobs) + return bytes; + + bytes = saturatingAdd(bytes, static_cast(blobs->capacity() * sizeof(PuffinBlob))); + for (const auto & blob : *blobs) + { + bytes = saturatingAdd( + {bytes, + static_cast(blob.type.size()), + static_cast(blob.compression_codec.size()), + static_cast(blob.fields.capacity() * sizeof(Int32)), + static_cast(FOOTER_BLOB_OVERHEAD)}); + for (const auto & [prop_key, value] : blob.properties) + bytes = saturatingAdd(bytes, saturatingAdd(static_cast(prop_key.size()), static_cast(value.size()))); + } + return bytes; +} + +void PuffinFilesCache::clearFooterMemoUnlocked() +{ + footer_memo.clear(); + footer_memo_bytes = 0; +} + +PuffinFilesCache::PuffinFilesCache( + const String & cache_policy, + size_t max_size_in_bytes, + size_t max_count, + double size_ratio) + : Base( + cache_policy, + CurrentMetrics::PuffinFilesCacheBytes, + CurrentMetrics::PuffinFilesCacheFiles, + max_size_in_bytes, + max_count, + size_ratio) + , log(getLogger("PuffinFilesCache")) + , footer_memo_max_count(max_count) + , footer_memo_max_bytes(max_size_in_bytes) +{ +} + +void PuffinFilesCache::clear() +{ + Base::clear(); + std::lock_guard lock(footer_mutex); + clearFooterMemoUnlocked(); +} + +void PuffinFilesCache::setMaxSizeInBytes(size_t max_size_in_bytes) +{ + Base::setMaxSizeInBytes(max_size_in_bytes); + std::lock_guard lock(footer_mutex); + footer_memo_max_bytes = max_size_in_bytes; + if (footer_memo_max_bytes == 0 || footer_memo_bytes > footer_memo_max_bytes) + clearFooterMemoUnlocked(); +} + +void PuffinFilesCache::setMaxCount(size_t max_count) +{ + Base::setMaxCount(max_count); + std::lock_guard lock(footer_mutex); + footer_memo_max_count = max_count; + if (footer_memo_max_count > 0 && footer_memo.size() > footer_memo_max_count) + clearFooterMemoUnlocked(); +} + +size_t PuffinFilesCache::footerMemoEntries() const +{ + std::lock_guard lock(footer_mutex); + return footer_memo.size(); +} + +UInt64 PuffinFilesCache::footerMemoBytes() const +{ + std::lock_guard lock(footer_mutex); + return footer_memo_bytes; +} + +String PuffinFilesCache::makeStorageIdentity(const IObjectStorage & object_storage) +{ + /// Include getDescription() (S3 endpoint, Azure account URL, Local path, ...) so two + /// backends with the same bucket/prefix on different hosts do not share cache entries. + return object_storage.getName() + "://" + object_storage.getDescription() + "/" + + object_storage.getObjectsNamespace() + "/" + object_storage.getCommonKeyPrefix(); +} + +std::optional PuffinFilesCache::tryCreateKey( + const String & storage_identity, + const String & file_path, + const String & etag, + Int64 content_offset, + Int64 content_size_in_bytes, + const String & referenced_data_file, + UInt64 expected_cardinality, + UInt64 data_file_record_count) +{ + if (etag.empty()) + return std::nullopt; + + return PuffinFilesCacheKey{ + storage_identity, + file_path, + etag, + content_offset, + content_size_in_bytes, + referenced_data_file, + expected_cardinality, + data_file_record_count}; +} + +std::optional PuffinFilesCache::tryCreateFooterKey( + const String & storage_identity, + const String & file_path, + const String & etag) +{ + if (etag.empty()) + return std::nullopt; + + return PuffinFooterCacheKey{storage_identity, file_path, etag}; +} + +void PuffinFilesCache::onEntryRemoval(const size_t weight_loss, const MappedPtr &) +{ + LOG_TRACE(log, "Puffin files cache eviction"); + ProfileEvents::increment(ProfileEvents::PuffinFilesCacheWeightLost, weight_loss); +} + +} diff --git a/src/Storages/ObjectStorage/DataLakes/PuffinFilesCache.h b/src/Storages/ObjectStorage/DataLakes/PuffinFilesCache.h new file mode 100644 index 000000000000..8feb8c743468 --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/PuffinFilesCache.h @@ -0,0 +1,322 @@ +#pragma once + +#include +#include + +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +namespace ProfileEvents +{ +extern const Event PuffinFilesCacheHits; +extern const Event PuffinFilesCacheMisses; +extern const Event PuffinFilesCacheWeightLost; +} + +namespace DB +{ + +class IObjectStorage; + +struct PuffinFilesCacheKey +{ + /// Distinguishes object-storage backends that share the same relative path (and possibly etag). + /// Built via `makeStorageIdentity` from storage type + description (endpoint) + namespace + prefix. + String storage_identity; + String file_path; + String etag; + Int64 content_offset = 0; + Int64 content_size_in_bytes = 0; + String referenced_data_file; + /// Manifest DV record_count / expected roaring cardinality. Included so a cache hit cannot + /// skip re-validation when a later request declares a different cardinality for the same slice. + UInt64 expected_cardinality = 0; + /// Data-file manifest record_count used to bound DV positions. Same rationale as cardinality. + UInt64 data_file_record_count = 0; + + bool operator==(const PuffinFilesCacheKey & other) const; + + /// Approximate bytes for key strings + fixed key fields (used in entry weight). + UInt64 approximateMemoryBytes() const; +}; + +struct PuffinFilesCacheKeyHash +{ + size_t operator()(const PuffinFilesCacheKey & key) const; +}; + +struct PuffinFilesCacheCell : private boost::noncopyable +{ + DataLakeObjectMetadata::ExcludedRowsPtr excluded_rows; + bool is_empty_deletion_vector = false; + UInt64 memory_bytes = 0; + + PuffinFilesCacheCell(DataLakeObjectMetadata::ExcludedRowsPtr excluded_rows_, UInt64 key_memory_bytes_); + + static UInt64 calculateMemorySize( + bool is_empty_deletion_vector_, + const DataLakeObjectMetadata::ExcludedRowsPtr & excluded_rows_, + UInt64 key_memory_bytes_); + +private: + /// Hash-map node + LRU list node + shared_ptr control block underestimates are absorbed here. + static constexpr size_t SIZE_IN_MEMORY_OVERHEAD = 256; +}; + +struct PuffinFilesCacheWeightFunction +{ + size_t operator()(const PuffinFilesCacheCell & cell) const; +}; + +/// File-level footer identity (shared by all DV slices in one coalesced Puffin). +struct PuffinFooterCacheKey +{ + String storage_identity; + String file_path; + String etag; + + bool operator==(const PuffinFooterCacheKey & other) const; +}; + +struct PuffinFooterCacheKeyHash +{ + size_t operator()(const PuffinFooterCacheKey & key) const; +}; + +/// Cache for parsed content loaded from Puffin files (deletion vectors today, indexes later). +/// Also memoizes parsed footers so coalesced multi-DV Puffins parse the footer once per file. +/// Footer memo shares `puffin_files_cache_size` / max-entry limits (size 0 disables memoization). +class PuffinFilesCache : public CacheBase +{ +public: + using Base = CacheBase; + using FooterBlobsPtr = std::shared_ptr>; + + PuffinFilesCache(const String & cache_policy, size_t max_size_in_bytes, size_t max_count, double size_ratio); + + /// Stable backend identity for cache keys: + /// `getName()://getDescription()/getObjectsNamespace()/getCommonKeyPrefix()`. + static String makeStorageIdentity(const IObjectStorage & object_storage); + + static std::optional tryCreateKey( + const String & storage_identity, + const String & file_path, + const String & etag, + Int64 content_offset, + Int64 content_size_in_bytes, + const String & referenced_data_file, + UInt64 expected_cardinality, + UInt64 data_file_record_count); + + static std::optional tryCreateFooterKey( + const String & storage_identity, + const String & file_path, + const String & etag); + + /// Clears deletion-vector entries and footer memo. + void clear(); + + void setMaxSizeInBytes(size_t max_size_in_bytes); + void setMaxCount(size_t max_count); + + /// Test/observability helpers for the footer memo. + size_t footerMemoEntries() const; + UInt64 footerMemoBytes() const; + + /// Small memo (not a weighted LRU): shares byte/count limits with the DV cache. + /// Concurrent misses on the same key each run `load_fn` (no stampede / waiter token); + /// only coalesced sequential slice loads share one parse. On insert, entries are dropped + /// one-by-one until the new entry fits — not a full memo clear. + template + FooterBlobsPtr getOrSetFooter(const PuffinFooterCacheKey & key, LoadFunc && load_fn) + { + { + std::lock_guard lock(footer_mutex); + if (footer_memo_max_bytes != 0) + { + if (auto it = footer_memo.find(key); it != footer_memo.end()) + return it->second.blobs; + } + } + + auto blobs = load_fn(); + + std::lock_guard lock(footer_mutex); + if (footer_memo_max_bytes == 0) + return blobs; + + if (auto it = footer_memo.find(key); it != footer_memo.end()) + return it->second.blobs; + + const UInt64 entry_bytes = approximateFooterEntryBytes(key, blobs); + /// A single entry larger than the whole budget is not memoized (caller still gets blobs). + if (entry_bytes > footer_memo_max_bytes) + return blobs; + + while ((footer_memo_max_count > 0 && footer_memo.size() >= footer_memo_max_count) + || footer_memo_bytes > footer_memo_max_bytes - entry_bytes) + { + if (footer_memo.empty()) + break; + + auto victim = footer_memo.begin(); + const UInt64 victim_bytes = victim->second.memory_bytes; + footer_memo.erase(victim); + if (footer_memo_bytes >= victim_bytes) + footer_memo_bytes -= victim_bytes; + else + footer_memo_bytes = 0; + } + + auto [it, inserted] = footer_memo.emplace(key, FooterMemoEntry{blobs, entry_bytes}); + if (inserted) + footer_memo_bytes += entry_bytes; + return it->second.blobs; + } + + template + DataLakeObjectMetadata::ExcludedRowsPtr getOrSetDeletionVector(const PuffinFilesCacheKey & key, LoadFunc && load_fn) + { + /// True if this caller's load_fn ran. Used only for tracing; hit/miss metrics use the + /// atomic CacheGetOrSetOutcome from getOrSetWithOutcome (not a follow-up contains()). + bool loaded = false; + auto load_fn_wrapper = [&]() + { + loaded = true; + auto excluded_rows = load_fn(); + const bool is_empty_deletion_vector = !excluded_rows; + if (is_empty_deletion_vector) + { + LOG_TRACE( + log, + "Cached empty puffin deletion vector for {} | {} | {} at offset {} length {} for data file {}", + key.storage_identity, + key.file_path, + key.etag, + key.content_offset, + key.content_size_in_bytes, + key.referenced_data_file); + } + else + { + LOG_TRACE( + log, + "Loaded puffin deletion vector into cache for {} | {} | {} at offset {} length {} for data file {}", + key.storage_identity, + key.file_path, + key.etag, + key.content_offset, + key.content_size_in_bytes, + key.referenced_data_file); + } + return std::make_shared(std::move(excluded_rows), key.approximateMemoryBytes()); + }; + + auto [cell, outcome] = Base::getOrSetWithOutcome(key, load_fn_wrapper); + const bool served_from_cache = outcome == CacheGetOrSetOutcome::Hit; + if (!served_from_cache) + { + if (loaded && outcome == CacheGetOrSetOutcome::MissNotResident) + { + LOG_TRACE( + log, + "Puffin files cache miss (load discarded by concurrent clear) for {} | {} | {} at offset {} length {} for data file {}", + key.storage_identity, + key.file_path, + key.etag, + key.content_offset, + key.content_size_in_bytes, + key.referenced_data_file); + } + else if (!loaded && outcome == CacheGetOrSetOutcome::MissNotResident) + { + LOG_TRACE( + log, + "Puffin files cache miss (waited for load discarded by concurrent clear) for {} | {} | {} at offset {} length {} for data file {}", + key.storage_identity, + key.file_path, + key.etag, + key.content_offset, + key.content_size_in_bytes, + key.referenced_data_file); + } + else + { + LOG_TRACE( + log, + "Puffin files cache miss for {} | {} | {} at offset {} length {} for data file {}", + key.storage_identity, + key.file_path, + key.etag, + key.content_offset, + key.content_size_in_bytes, + key.referenced_data_file); + } + ProfileEvents::increment(ProfileEvents::PuffinFilesCacheMisses); + } + else if (cell->is_empty_deletion_vector) + { + LOG_TRACE( + log, + "Puffin files cache hit (empty deletion vector) for {} | {} | {} at offset {} length {} for data file {}", + key.storage_identity, + key.file_path, + key.etag, + key.content_offset, + key.content_size_in_bytes, + key.referenced_data_file); + ProfileEvents::increment(ProfileEvents::PuffinFilesCacheHits); + } + else + { + LOG_TRACE( + log, + "Puffin files cache hit for {} | {} | {} at offset {} length {} for data file {}", + key.storage_identity, + key.file_path, + key.etag, + key.content_offset, + key.content_size_in_bytes, + key.referenced_data_file); + ProfileEvents::increment(ProfileEvents::PuffinFilesCacheHits); + } + + return cloneExcludedRows(*cell); + } + +private: + struct FooterMemoEntry + { + FooterBlobsPtr blobs; + UInt64 memory_bytes = 0; + }; + + static DataLakeObjectMetadata::ExcludedRowsPtr cloneExcludedRows(const PuffinFilesCacheCell & cell); + static UInt64 approximateFooterEntryBytes(const PuffinFooterCacheKey & key, const FooterBlobsPtr & blobs); + + void clearFooterMemoUnlocked(); + + LoggerPtr log; + mutable std::mutex footer_mutex; + std::unordered_map footer_memo; + size_t footer_memo_max_count = 0; + size_t footer_memo_max_bytes = 0; + UInt64 footer_memo_bytes = 0; + + void onEntryRemoval(size_t weight_loss, const MappedPtr &) override; +}; + +using PuffinFilesCachePtr = std::shared_ptr; + +} diff --git a/src/Storages/ObjectStorage/DataLakes/tests/gtest_deletion_vector_before_equality_filter.cpp b/src/Storages/ObjectStorage/DataLakes/tests/gtest_deletion_vector_before_equality_filter.cpp new file mode 100644 index 000000000000..3b379381ca76 --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/tests/gtest_deletion_vector_before_equality_filter.cpp @@ -0,0 +1,86 @@ +#include + +#include +#include +#include +#include + +using namespace DB; + +namespace +{ + +Chunk makeChunkWithFileRowNumbers(const std::vector & values, size_t row_num_offset = 0) +{ + auto column = ColumnUInt64::create(); + for (UInt64 value : values) + column->insert(value); + + Chunk chunk(Columns{std::move(column)}, values.size()); + chunk.getChunkInfos().add(std::make_shared(row_num_offset)); + return chunk; +} + +/// Mimics Iceberg equality-delete `FilterTransform`: shrink columns, leave `applied_filter` unset. +void shrinkWithoutAppliedFilter(Chunk & chunk, const IColumn::Filter & filter) +{ + size_t result_size = 0; + for (UInt8 keep : filter) + result_size += keep != 0; + + auto columns = chunk.detachColumns(); + for (auto & column : columns) + column = column->filter(filter, -1); + chunk.setColumns(std::move(columns), result_size); +} + +std::vector readValues(const Chunk & chunk) +{ + const auto & column = assert_cast(*chunk.getColumns().at(0)); + std::vector values; + values.reserve(column.size()); + for (size_t i = 0; i < column.size(); ++i) + values.push_back(column.getData()[i]); + return values; +} + +std::shared_ptr makeExcludedRows(std::initializer_list positions) +{ + auto excluded_rows = std::make_shared(); + for (UInt64 position : positions) + excluded_rows->add(position); + return excluded_rows; +} + +} + +/// Equality removes file row 1; DV deletes file position 2. Survivors must be {0, 3}. +TEST(DeletionVectorBeforeEqualityFilter, CorrectOrderKeepsFileRowMapping) +{ + Chunk chunk = makeChunkWithFileRowNumbers({0, 1, 2, 3}); + + DeletionVectorTransform::transform(chunk, *makeExcludedRows({2})); + ASSERT_EQ(readValues(chunk), (std::vector{0, 1, 3})); + + /// Drop equality-deleted value 1 without updating applied_filter (as FilterTransform does). + shrinkWithoutAppliedFilter(chunk, IColumn::Filter{1, 0, 1}); + EXPECT_EQ(readValues(chunk), (std::vector{0, 3})); +} + +/// Documents why StorageObjectStorageSource must run DV before equality FilterTransform: +/// reversing the order maps DV positions onto dense post-equality indices and keeps the +/// wrong survivors ({0, 2} instead of {0, 3}). +TEST(DeletionVectorBeforeEqualityFilter, WrongOrderProducesWrongSurvivors) +{ + Chunk chunk = makeChunkWithFileRowNumbers({0, 1, 2, 3}); + + /// Equality first: drop value 1 → dense [0, 2, 3], applied_filter unset. + shrinkWithoutAppliedFilter(chunk, IColumn::Filter{1, 0, 1, 1}); + ASSERT_EQ(readValues(chunk), (std::vector{0, 2, 3})); + + /// DV still targets file position 2, but ChunkInfoRowNumbers still starts at 0 over the + /// shrunk chunk, so index 2 (value 3) is removed instead of original file row 2. + DeletionVectorTransform::transform(chunk, *makeExcludedRows({2})); + EXPECT_EQ(readValues(chunk), (std::vector{0, 2})); + EXPECT_NE(readValues(chunk), (std::vector{0, 3})); +} diff --git a/src/Storages/ObjectStorage/DataLakes/tests/gtest_deletion_vector_need_only_count.cpp b/src/Storages/ObjectStorage/DataLakes/tests/gtest_deletion_vector_need_only_count.cpp new file mode 100644 index 000000000000..b943fe568fc6 --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/tests/gtest_deletion_vector_need_only_count.cpp @@ -0,0 +1,161 @@ +#include + +#include +#include +#include +#include +#include +#include + +using namespace DB; + +namespace +{ + +Chunk makeConstCountChunk(size_t num_rows, size_t row_num_offset) +{ + auto nested = ColumnUInt64::create(); + nested->insertDefault(); + Columns columns; + columns.emplace_back(ColumnConst::create(std::move(nested), num_rows)); + Chunk chunk(std::move(columns), num_rows); + chunk.getChunkInfos().add(std::make_shared(row_num_offset)); + return chunk; +} + +Chunk makeMaterializedChunk(const std::vector & values, size_t row_num_offset = 0) +{ + auto column = ColumnUInt64::create(); + for (UInt64 value : values) + column->insert(value); + + Chunk chunk(Columns{std::move(column)}, values.size()); + chunk.getChunkInfos().add(std::make_shared(row_num_offset)); + return chunk; +} + +std::shared_ptr makeExcludedRows(std::initializer_list positions) +{ + auto excluded_rows = std::make_shared(); + for (UInt64 position : positions) + excluded_rows->add(position); + return excluded_rows; +} + +} + +TEST(RoaringBitmapRangeCardinality, CountsInclusiveStartExclusiveEnd) +{ + DataLakeObjectMetadata::ExcludedRows bitmap; + bitmap.add(1); + bitmap.add(5); + bitmap.add(10); + + EXPECT_EQ(bitmap.rb_range_cardinality(0, 0), 0u); + EXPECT_EQ(bitmap.rb_range_cardinality(5, 5), 0u); + EXPECT_EQ(bitmap.rb_range_cardinality(0, 5), 1u); + EXPECT_EQ(bitmap.rb_range_cardinality(0, 6), 2u); + EXPECT_EQ(bitmap.rb_range_cardinality(1, 11), 3u); + EXPECT_EQ(bitmap.rb_range_cardinality(2, 10), 1u); + EXPECT_EQ(bitmap.rb_range_cardinality(11, 100), 0u); +} + +TEST(RoaringBitmapRangeCardinality, MatchesRbRangeCountForLargeBitmap) +{ + DataLakeObjectMetadata::ExcludedRows bitmap; + /// Force large roaring path (small-set threshold is 32). + for (UInt64 i = 0; i < 64; ++i) + bitmap.add(i * 3); + ASSERT_TRUE(bitmap.isLarge()); + + DataLakeObjectMetadata::ExcludedRows subset; + const UInt64 via_range = bitmap.rb_range(10, 100, subset); + EXPECT_EQ(bitmap.rb_range_cardinality(10, 100), via_range); + EXPECT_EQ(subset.size(), via_range); +} + +TEST(RoaringBitmapRangeCardinality, SumOfRowGroupRangesMatchesWholeFile) +{ + DataLakeObjectMetadata::ExcludedRows bitmap; + for (UInt64 i = 0; i < 10'000; ++i) + bitmap.add(i * 7); + ASSERT_TRUE(bitmap.isLarge()); + + constexpr size_t rows_per_group = 1'000; + constexpr size_t num_groups = 200; + UInt64 summed = 0; + for (size_t group = 0; group < num_groups; ++group) + { + const UInt64 start = group * rows_per_group; + summed += bitmap.rb_range_cardinality(start, start + rows_per_group); + } + + EXPECT_EQ(summed, bitmap.rb_range_cardinality(0, num_groups * rows_per_group)); + EXPECT_EQ(summed, bitmap.size()); +} + +TEST(DeletionVectorNeedOnlyCount, ConstChunkUsesRangeCardinality) +{ + /// Large enough that a dense column Filter would be expensive; const columns stay O(1) via + /// cloneResized. When deletes land in-range we still record `applied_filter` (O(N) mask) so a + /// later row-number consumer can map dense indices back to file rows. + constexpr size_t num_rows = 5'000'000; + Chunk chunk = makeConstCountChunk(num_rows, /*row_num_offset=*/0); + + DeletionVectorTransform::transform(chunk, *makeExcludedRows({0, 1, num_rows - 1, num_rows + 10})); + EXPECT_EQ(chunk.getNumRows(), num_rows - 3); + ASSERT_EQ(chunk.getNumColumns(), 1u); + EXPECT_TRUE(isColumnConst(*chunk.getColumns()[0])); + + const auto chunk_info = chunk.getChunkInfos().get(); + ASSERT_TRUE(chunk_info->applied_filter.has_value()); + const auto & filter = chunk_info->applied_filter.value(); + ASSERT_EQ(filter.size(), num_rows); + EXPECT_EQ(filter[0], 0); + EXPECT_EQ(filter[1], 0); + EXPECT_EQ(filter[num_rows - 1], 0); + EXPECT_EQ(filter[2], 1); + + size_t kept = 0; + for (UInt8 bit : filter) + kept += bit != 0; + EXPECT_EQ(kept, chunk.getNumRows()); +} + +TEST(DeletionVectorNeedOnlyCount, ConstChunkHonorsRowNumOffset) +{ + Chunk chunk = makeConstCountChunk(/*num_rows=*/10, /*row_num_offset=*/100); + /// Deletes at absolute positions 99 (before), 105 (inside), 110 (end exclusive / outside). + DeletionVectorTransform::transform(chunk, *makeExcludedRows({99, 105, 110})); + EXPECT_EQ(chunk.getNumRows(), 9u); + + const auto chunk_info = chunk.getChunkInfos().get(); + ASSERT_TRUE(chunk_info->applied_filter.has_value()); + const auto & filter = chunk_info->applied_filter.value(); + ASSERT_EQ(filter.size(), 10u); + EXPECT_EQ(filter[5], 0); + EXPECT_EQ(filter[0], 1); +} + +TEST(DeletionVectorNeedOnlyCount, ConstChunkThenSecondBitmapUsesAppliedFilter) +{ + /// Simulates DV const-count path followed by another DeletionVectorTransform (as + /// IcebergBitmapPositionDeleteTransform does). Without applied_filter the second pass would + /// treat the shrunk const chunk as consecutive file rows and delete the wrong positions. + Chunk chunk = makeConstCountChunk(/*num_rows=*/8, /*row_num_offset=*/10); + DeletionVectorTransform::transform(chunk, *makeExcludedRows({11, 14})); + EXPECT_EQ(chunk.getNumRows(), 6u); + ASSERT_TRUE(chunk.getChunkInfos().get()->applied_filter.has_value()); + + /// Delete absolute file row 16 (dense survivor index 4 after the first pass). + DeletionVectorTransform::transform(chunk, *makeExcludedRows({16})); + EXPECT_EQ(chunk.getNumRows(), 5u); +} + +TEST(DeletionVectorNeedOnlyCount, MaterializedChunkStillUsesDenseFilter) +{ + Chunk chunk = makeMaterializedChunk({10, 11, 12, 13}, /*row_num_offset=*/10); + DeletionVectorTransform::transform(chunk, *makeExcludedRows({11, 13})); + EXPECT_EQ(chunk.getNumRows(), 2u); + ASSERT_TRUE(chunk.getChunkInfos().get()->applied_filter.has_value()); +} diff --git a/src/Storages/ObjectStorage/DataLakes/tests/gtest_has_non_empty_excluded_rows.cpp b/src/Storages/ObjectStorage/DataLakes/tests/gtest_has_non_empty_excluded_rows.cpp new file mode 100644 index 000000000000..9903493c3413 --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/tests/gtest_has_non_empty_excluded_rows.cpp @@ -0,0 +1,28 @@ +#include + +#include +#include + +using namespace DB; + +TEST(HasNonEmptyExcludedRows, EmptyOptionalIsFalse) +{ + EXPECT_FALSE(hasNonEmptyExcludedRows(std::nullopt)); +} + +TEST(HasNonEmptyExcludedRows, MissingOrEmptyBitmapIsFalse) +{ + DataLakeObjectMetadata metadata; + EXPECT_FALSE(hasNonEmptyExcludedRows(metadata)); + + metadata.excluded_rows = std::make_shared(); + EXPECT_FALSE(hasNonEmptyExcludedRows(metadata)); +} + +TEST(HasNonEmptyExcludedRows, NonEmptyBitmapIsTrue) +{ + DataLakeObjectMetadata metadata; + metadata.excluded_rows = std::make_shared(); + metadata.excluded_rows->add(7); + EXPECT_TRUE(hasNonEmptyExcludedRows(metadata)); +} diff --git a/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_deletion_vector_bounds.cpp b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_deletion_vector_bounds.cpp new file mode 100644 index 000000000000..0c036309b73b --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_deletion_vector_bounds.cpp @@ -0,0 +1,63 @@ +#include + +#include +#include +#include + +using namespace DB; + +namespace DB +{ +namespace ErrorCodes +{ +extern const int BAD_ARGUMENTS; +} +} + +TEST(PuffinDeletionVectorBounds, RejectsLengthExceedingFileSize) +{ + const String data(64, '\0'); + ReadBufferFromOutsideMemoryFile file("test.puffin", data); + + try + { + readDeletionVectorFromPuffin(file, 0, 1000); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::BAD_ARGUMENTS); + EXPECT_NE(e.message().find("offset/length out of bounds"), std::string::npos); + } +} + +TEST(PuffinDeletionVectorBounds, RejectsOffsetPlusLengthOverflow) +{ + const String data(64, '\0'); + ReadBufferFromOutsideMemoryFile file("test.puffin", data); + + try + { + readDeletionVectorFromPuffin(file, 60, 12); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::BAD_ARGUMENTS); + EXPECT_NE(e.message().find("offset/length out of bounds"), std::string::npos); + } +} + +TEST(PuffinDeletionVectorBounds, ValidatePuffinBlobBoundsRejectsNegativeOffset) +{ + try + { + validatePuffinBlobBounds(-1, 10, 64); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::BAD_ARGUMENTS); + EXPECT_NE(e.message().find("offset/length out of bounds"), std::string::npos); + } +} diff --git a/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_deletion_vector_cardinality.cpp b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_deletion_vector_cardinality.cpp new file mode 100644 index 000000000000..2253f4ac548d --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_deletion_vector_cardinality.cpp @@ -0,0 +1,96 @@ +#include + +#include +#include +#include + +#include + +using namespace DB; + +namespace DB +{ +namespace ErrorCodes +{ +extern const int BAD_ARGUMENTS; +} +} + +namespace +{ + +/// deletion-vector-v1 blob for positions {2, 5} (cardinality 2). +constexpr UInt8 two_position_dv_blob[] = { + 0x00, 0x00, 0x00, 0x24, 0xD1, 0xD3, 0x39, 0x64, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x3A, 0x30, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, + 0x10, 0x00, 0x00, 0x00, 0x02, 0x00, 0x05, 0x00, 0x2C, 0xDB, 0x9F, 0xC1, +}; + +constexpr Int64 large_declared_length = 64 * 1024 * 1024; + +} + +TEST(PuffinDeletionVectorCardinality, RejectsCardinalityAboveMaterializationLimitBeforeParse) +{ + /// Ceiling is checked before payload validation, so even an empty blob must fail closed. + try + { + deserializeDeletionVectorV1Blob(std::string_view{}, PUFFIN_DV_MAX_MATERIALIZED_POSITIONS + 1); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::BAD_ARGUMENTS); + EXPECT_NE(e.message().find("exceeds materialization limit"), std::string::npos); + } +} + +TEST(PuffinDeletionVectorCardinality, RejectsCardinalityAboveLimitBeforeFullAllocate) +{ + /// Huge declared length with only a tiny buffer: without an early ceiling check this would + /// allocate `large_declared_length` (or fail mid-read after that allocate). ReadBufferFromMemory + /// does not expose file size, so bounds checks alone do not stop this. + /// The SQL `Puffin` format applies the same early check in `readDeletionVectorBlobBytes`. + const char header[8] = {}; + ReadBufferFromMemory file(header, sizeof(header)); + + try + { + readDeletionVectorFromPuffin( + file, 0, large_declared_length, PUFFIN_DV_MAX_MATERIALIZED_POSITIONS + 1); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::BAD_ARGUMENTS); + EXPECT_NE(e.message().find("exceeds materialization limit"), std::string::npos); + } +} + +TEST(PuffinDeletionVectorCardinality, RejectsBitmapExceedingDeclaredCardinality) +{ + const std::string_view blob( + reinterpret_cast(two_position_dv_blob), sizeof(two_position_dv_blob)); + + try + { + deserializeDeletionVectorV1Blob(blob, /*expected_cardinality=*/1); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::BAD_ARGUMENTS); + EXPECT_NE(e.message().find("exceeds declared cardinality"), std::string::npos); + } +} + +TEST(PuffinDeletionVectorCardinality, AcceptsMatchingCardinality) +{ + const std::string_view blob( + reinterpret_cast(two_position_dv_blob), sizeof(two_position_dv_blob)); + + const auto positions = deserializeDeletionVectorV1Blob(blob, /*expected_cardinality=*/2); + ASSERT_EQ(positions.size(), 2u); + EXPECT_EQ(positions[0], 2u); + EXPECT_EQ(positions[1], 5u); +} diff --git a/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_deletion_vector_envelope.cpp b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_deletion_vector_envelope.cpp new file mode 100644 index 000000000000..4db35bad5ad4 --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_deletion_vector_envelope.cpp @@ -0,0 +1,124 @@ +#include + +#include +#include +#include + +#include + +using namespace DB; + +namespace DB +{ +namespace ErrorCodes +{ +extern const int BAD_ARGUMENTS; +} +} + +namespace +{ + +/// deletion-vector-v1 blob for positions {2, 5} (cardinality 2). +constexpr UInt8 two_position_dv_blob[] = { + 0x00, 0x00, 0x00, 0x24, 0xD1, 0xD3, 0x39, 0x64, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x3A, 0x30, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, + 0x10, 0x00, 0x00, 0x00, 0x02, 0x00, 0x05, 0x00, 0x2C, 0xDB, 0x9F, 0xC1, +}; + +/// Large declared length that would force a huge allocate if peeked after full read. +constexpr Int64 large_declared_length = 64 * 1024 * 1024; + +} + +TEST(PuffinDeletionVectorEnvelope, RejectsLengthBelowEnvelopeMinimum) +{ + const char zeros[16] = {}; + ReadBufferFromMemory file(zeros, sizeof(zeros)); + + try + { + readDeletionVectorFromPuffin(file, 0, 11); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::BAD_ARGUMENTS); + EXPECT_NE(e.message().find("too small"), std::string::npos); + } +} + +TEST(PuffinDeletionVectorEnvelope, RejectsInvalidMagicBeforeFullAllocate) +{ + /// Only 8 bytes available; declared length is huge. Without envelope peek this would allocate + /// `large_declared_length` (or fail mid-read after that allocate). ReadBufferFromMemory does not + /// expose file size, so the absolute 2 GiB / bounds checks alone do not stop this. + const UInt8 header[8] = {0x00, 0x00, 0x00, 0x24, 0x00, 0x00, 0x00, 0x00}; // wrong magic + ReadBufferFromMemory file(header, sizeof(header)); + + try + { + readDeletionVectorFromPuffin(file, 0, large_declared_length); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::BAD_ARGUMENTS); + EXPECT_NE(e.message().find("Invalid deletion vector magic"), std::string::npos); + } +} + +TEST(PuffinDeletionVectorEnvelope, RejectsCombinedLengthMismatchBeforeFullAllocate) +{ + /// Valid magic, but combined_length implies blob size 0x24+8=44, while caller length is huge. + const UInt8 header[8] = {0x00, 0x00, 0x00, 0x24, 0xD1, 0xD3, 0x39, 0x64}; + ReadBufferFromMemory file(header, sizeof(header)); + + try + { + readDeletionVectorFromPuffin(file, 0, large_declared_length); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::BAD_ARGUMENTS); + EXPECT_NE(e.message().find("does not match combined length"), std::string::npos); + } +} + +TEST(PuffinDeletionVectorEnvelope, ReadsValidBlobAfterEnvelopePeek) +{ + ReadBufferFromMemory file(two_position_dv_blob, sizeof(two_position_dv_blob)); + const auto positions = readDeletionVectorFromPuffin( + file, 0, static_cast(sizeof(two_position_dv_blob)), /*expected_cardinality=*/2); + + ASSERT_EQ(positions.size(), 2u); + EXPECT_EQ(positions[0], 2u); + EXPECT_EQ(positions[1], 5u); +} + +TEST(PuffinDeletionVectorEnvelope, RejectsInternallyInconsistentRoaringAfterReadSafe) +{ + /// Valid DV envelope + CRC wrapping a portable roaring that `readSafe` accepts but + /// `roaring_bitmap_internal_validate` rejects (duplicate values in an array container). + /// Bytes from croaring's robust_deserialization_unit `deserialize_unsorted_array` fixture. + constexpr UInt8 inconsistent_roaring_dv_blob[] = { + 0x00, 0x00, 0x00, 0x1D, 0xD1, 0xD3, 0x39, 0x64, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x3B, 0x30, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x6E, 0x0E, 0x9B, 0x12, + }; + + const std::string_view blob( + reinterpret_cast(inconsistent_roaring_dv_blob), sizeof(inconsistent_roaring_dv_blob)); + + try + { + deserializeDeletionVectorV1Blob(blob, /*expected_cardinality=*/2); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::BAD_ARGUMENTS); + EXPECT_NE(e.message().find("failed internal validation"), std::string::npos); + } +} diff --git a/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_deletion_vector_footer_bind.cpp b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_deletion_vector_footer_bind.cpp new file mode 100644 index 000000000000..66dcd1e7ad8f --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_deletion_vector_footer_bind.cpp @@ -0,0 +1,258 @@ +#include + +#include +#include +#include + +using namespace DB; + +namespace DB +{ +namespace ErrorCodes +{ +extern const int BAD_ARGUMENTS; +} +} + +namespace +{ + +/// Two equal-cardinality deletion-vector-v1 blobs for different data files. +/// Blob A at offset 4 length 44 -> /data/file_a.parquet positions {2, 5} +/// Blob B at offset 48 length 44 -> /data/file_b.parquet positions {7, 9} +constexpr UInt8 two_equal_cardinality_dvs_puffin[] = { + 0x50, 0x46, 0x41, 0x31, 0x00, 0x00, 0x00, 0x24, 0xD1, 0xD3, 0x39, 0x64, 0x01, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3A, 0x30, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x01, 0x00, 0x10, 0x00, 0x00, 0x00, 0x02, 0x00, 0x05, 0x00, 0x2C, 0xDB, 0x9F, 0xC1, + 0x00, 0x00, 0x00, 0x24, 0xD1, 0xD3, 0x39, 0x64, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x3A, 0x30, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, + 0x10, 0x00, 0x00, 0x00, 0x07, 0x00, 0x09, 0x00, 0xB7, 0xB0, 0x20, 0xFF, 0x50, 0x46, 0x41, 0x31, + 0x7B, 0x22, 0x62, 0x6C, 0x6F, 0x62, 0x73, 0x22, 0x3A, 0x20, 0x5B, 0x7B, 0x22, 0x74, 0x79, 0x70, + 0x65, 0x22, 0x3A, 0x20, 0x22, 0x64, 0x65, 0x6C, 0x65, 0x74, 0x69, 0x6F, 0x6E, 0x2D, 0x76, 0x65, + 0x63, 0x74, 0x6F, 0x72, 0x2D, 0x76, 0x31, 0x22, 0x2C, 0x20, 0x22, 0x66, 0x69, 0x65, 0x6C, 0x64, + 0x73, 0x22, 0x3A, 0x20, 0x5B, 0x5D, 0x2C, 0x20, 0x22, 0x73, 0x6E, 0x61, 0x70, 0x73, 0x68, 0x6F, + 0x74, 0x2D, 0x69, 0x64, 0x22, 0x3A, 0x20, 0x2D, 0x31, 0x2C, 0x20, 0x22, 0x73, 0x65, 0x71, 0x75, + 0x65, 0x6E, 0x63, 0x65, 0x2D, 0x6E, 0x75, 0x6D, 0x62, 0x65, 0x72, 0x22, 0x3A, 0x20, 0x2D, 0x31, + 0x2C, 0x20, 0x22, 0x6F, 0x66, 0x66, 0x73, 0x65, 0x74, 0x22, 0x3A, 0x20, 0x34, 0x2C, 0x20, 0x22, + 0x6C, 0x65, 0x6E, 0x67, 0x74, 0x68, 0x22, 0x3A, 0x20, 0x34, 0x34, 0x2C, 0x20, 0x22, 0x70, 0x72, + 0x6F, 0x70, 0x65, 0x72, 0x74, 0x69, 0x65, 0x73, 0x22, 0x3A, 0x20, 0x7B, 0x22, 0x72, 0x65, 0x66, + 0x65, 0x72, 0x65, 0x6E, 0x63, 0x65, 0x64, 0x2D, 0x64, 0x61, 0x74, 0x61, 0x2D, 0x66, 0x69, 0x6C, + 0x65, 0x22, 0x3A, 0x20, 0x22, 0x2F, 0x64, 0x61, 0x74, 0x61, 0x2F, 0x66, 0x69, 0x6C, 0x65, 0x5F, + 0x61, 0x2E, 0x70, 0x61, 0x72, 0x71, 0x75, 0x65, 0x74, 0x22, 0x2C, 0x20, 0x22, 0x63, 0x61, 0x72, + 0x64, 0x69, 0x6E, 0x61, 0x6C, 0x69, 0x74, 0x79, 0x22, 0x3A, 0x20, 0x22, 0x32, 0x22, 0x7D, 0x7D, + 0x2C, 0x20, 0x7B, 0x22, 0x74, 0x79, 0x70, 0x65, 0x22, 0x3A, 0x20, 0x22, 0x64, 0x65, 0x6C, 0x65, + 0x74, 0x69, 0x6F, 0x6E, 0x2D, 0x76, 0x65, 0x63, 0x74, 0x6F, 0x72, 0x2D, 0x76, 0x31, 0x22, 0x2C, + 0x20, 0x22, 0x66, 0x69, 0x65, 0x6C, 0x64, 0x73, 0x22, 0x3A, 0x20, 0x5B, 0x5D, 0x2C, 0x20, 0x22, + 0x73, 0x6E, 0x61, 0x70, 0x73, 0x68, 0x6F, 0x74, 0x2D, 0x69, 0x64, 0x22, 0x3A, 0x20, 0x2D, 0x31, + 0x2C, 0x20, 0x22, 0x73, 0x65, 0x71, 0x75, 0x65, 0x6E, 0x63, 0x65, 0x2D, 0x6E, 0x75, 0x6D, 0x62, + 0x65, 0x72, 0x22, 0x3A, 0x20, 0x2D, 0x31, 0x2C, 0x20, 0x22, 0x6F, 0x66, 0x66, 0x73, 0x65, 0x74, + 0x22, 0x3A, 0x20, 0x34, 0x38, 0x2C, 0x20, 0x22, 0x6C, 0x65, 0x6E, 0x67, 0x74, 0x68, 0x22, 0x3A, + 0x20, 0x34, 0x34, 0x2C, 0x20, 0x22, 0x70, 0x72, 0x6F, 0x70, 0x65, 0x72, 0x74, 0x69, 0x65, 0x73, + 0x22, 0x3A, 0x20, 0x7B, 0x22, 0x72, 0x65, 0x66, 0x65, 0x72, 0x65, 0x6E, 0x63, 0x65, 0x64, 0x2D, + 0x64, 0x61, 0x74, 0x61, 0x2D, 0x66, 0x69, 0x6C, 0x65, 0x22, 0x3A, 0x20, 0x22, 0x2F, 0x64, 0x61, + 0x74, 0x61, 0x2F, 0x66, 0x69, 0x6C, 0x65, 0x5F, 0x62, 0x2E, 0x70, 0x61, 0x72, 0x71, 0x75, 0x65, + 0x74, 0x22, 0x2C, 0x20, 0x22, 0x63, 0x61, 0x72, 0x64, 0x69, 0x6E, 0x61, 0x6C, 0x69, 0x74, 0x79, + 0x22, 0x3A, 0x20, 0x22, 0x32, 0x22, 0x7D, 0x7D, 0x5D, 0x7D, 0x9A, 0x01, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x50, 0x46, 0x41, 0x31, +}; + +std::vector readFixtureBlobs() +{ + ReadBufferFromMemory file(two_equal_cardinality_dvs_puffin, sizeof(two_equal_cardinality_dvs_puffin)); + return readPuffinFooterBlobsFromSeekable(file, sizeof(two_equal_cardinality_dvs_puffin)); +} + +} + +TEST(PuffinDeletionVectorFooterBind, AcceptsMatchingOffsetReferencedFileAndCardinality) +{ + const auto blobs = readFixtureBlobs(); + const auto & blob = bindDeletionVectorBlob(blobs, /*content_offset=*/4, /*content_size_in_bytes=*/44, "/data/file_a.parquet", 2); + EXPECT_EQ(blob.type, "deletion-vector-v1"); + EXPECT_EQ(blob.properties.at("referenced-data-file"), "/data/file_a.parquet"); + + ReadBufferFromMemory file(two_equal_cardinality_dvs_puffin, sizeof(two_equal_cardinality_dvs_puffin)); + const auto positions = readDeletionVectorFromPuffin(file, 4, 44, /*expected_cardinality=*/2); + ASSERT_EQ(positions.size(), 2u); + EXPECT_EQ(positions[0], 2u); + EXPECT_EQ(positions[1], 5u); +} + +TEST(PuffinDeletionVectorFooterBind, RejectsSwappedOffsetForOtherFilesEqualCardinalityBlob) +{ + /// Manifest for file A points at file B's slice (same length and cardinality). + const auto blobs = readFixtureBlobs(); + try + { + bindDeletionVectorBlob(blobs, /*content_offset=*/48, /*content_size_in_bytes=*/44, "/data/file_a.parquet", 2); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::BAD_ARGUMENTS); + EXPECT_NE(e.message().find("referenced-data-file"), std::string::npos); + } +} + +TEST(PuffinDeletionVectorFooterBind, RejectsCardinalityMismatch) +{ + const auto blobs = readFixtureBlobs(); + try + { + bindDeletionVectorBlob(blobs, /*content_offset=*/4, /*content_size_in_bytes=*/44, "/data/file_a.parquet", 3); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::BAD_ARGUMENTS); + EXPECT_NE(e.message().find("cardinality"), std::string::npos); + } +} + +TEST(PuffinDeletionVectorFooterBind, RejectsMissingOffset) +{ + const auto blobs = readFixtureBlobs(); + try + { + bindDeletionVectorBlob(blobs, /*content_offset=*/99, /*content_size_in_bytes=*/44, "/data/file_a.parquet", 2); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::BAD_ARGUMENTS); + EXPECT_NE(e.message().find("No Puffin footer blob"), std::string::npos); + } +} + +namespace +{ + +/// Sketch placeholder (16 bytes) then deletion-vector-v1 for positions {2, 5} (44 bytes). +/// Keep in sync with tests/queries/0_stateless/data_puffin/mixed_blob_types.puffin. +constexpr UInt8 mixed_sketch_and_dv_puffin[] = { + 0x50, 0x46, 0x41, 0x31, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0xD1, 0xD3, 0x39, 0x64, 0x01, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3A, 0x30, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x01, 0x00, 0x10, 0x00, 0x00, 0x00, 0x02, 0x00, 0x05, 0x00, 0x2C, 0xDB, 0x9F, 0xC1, + 0x50, 0x46, 0x41, 0x31, 0x7B, 0x22, 0x62, 0x6C, 0x6F, 0x62, 0x73, 0x22, 0x3A, 0x20, 0x5B, 0x7B, + 0x22, 0x74, 0x79, 0x70, 0x65, 0x22, 0x3A, 0x20, 0x22, 0x61, 0x70, 0x61, 0x63, 0x68, 0x65, 0x2D, + 0x64, 0x61, 0x74, 0x61, 0x73, 0x6B, 0x65, 0x74, 0x63, 0x68, 0x65, 0x73, 0x2D, 0x74, 0x68, 0x65, + 0x74, 0x61, 0x2D, 0x76, 0x31, 0x22, 0x2C, 0x20, 0x22, 0x66, 0x69, 0x65, 0x6C, 0x64, 0x73, 0x22, + 0x3A, 0x20, 0x5B, 0x5D, 0x2C, 0x20, 0x22, 0x73, 0x6E, 0x61, 0x70, 0x73, 0x68, 0x6F, 0x74, 0x2D, + 0x69, 0x64, 0x22, 0x3A, 0x20, 0x2D, 0x31, 0x2C, 0x20, 0x22, 0x73, 0x65, 0x71, 0x75, 0x65, 0x6E, + 0x63, 0x65, 0x2D, 0x6E, 0x75, 0x6D, 0x62, 0x65, 0x72, 0x22, 0x3A, 0x20, 0x2D, 0x31, 0x2C, 0x20, + 0x22, 0x6F, 0x66, 0x66, 0x73, 0x65, 0x74, 0x22, 0x3A, 0x20, 0x34, 0x2C, 0x20, 0x22, 0x6C, 0x65, + 0x6E, 0x67, 0x74, 0x68, 0x22, 0x3A, 0x20, 0x31, 0x36, 0x2C, 0x20, 0x22, 0x70, 0x72, 0x6F, 0x70, + 0x65, 0x72, 0x74, 0x69, 0x65, 0x73, 0x22, 0x3A, 0x20, 0x7B, 0x7D, 0x7D, 0x2C, 0x20, 0x7B, 0x22, + 0x74, 0x79, 0x70, 0x65, 0x22, 0x3A, 0x20, 0x22, 0x64, 0x65, 0x6C, 0x65, 0x74, 0x69, 0x6F, 0x6E, + 0x2D, 0x76, 0x65, 0x63, 0x74, 0x6F, 0x72, 0x2D, 0x76, 0x31, 0x22, 0x2C, 0x20, 0x22, 0x66, 0x69, + 0x65, 0x6C, 0x64, 0x73, 0x22, 0x3A, 0x20, 0x5B, 0x5D, 0x2C, 0x20, 0x22, 0x73, 0x6E, 0x61, 0x70, + 0x73, 0x68, 0x6F, 0x74, 0x2D, 0x69, 0x64, 0x22, 0x3A, 0x20, 0x2D, 0x31, 0x2C, 0x20, 0x22, 0x73, + 0x65, 0x71, 0x75, 0x65, 0x6E, 0x63, 0x65, 0x2D, 0x6E, 0x75, 0x6D, 0x62, 0x65, 0x72, 0x22, 0x3A, + 0x20, 0x2D, 0x31, 0x2C, 0x20, 0x22, 0x6F, 0x66, 0x66, 0x73, 0x65, 0x74, 0x22, 0x3A, 0x20, 0x32, + 0x30, 0x2C, 0x20, 0x22, 0x6C, 0x65, 0x6E, 0x67, 0x74, 0x68, 0x22, 0x3A, 0x20, 0x34, 0x34, 0x2C, + 0x20, 0x22, 0x70, 0x72, 0x6F, 0x70, 0x65, 0x72, 0x74, 0x69, 0x65, 0x73, 0x22, 0x3A, 0x20, 0x7B, + 0x22, 0x72, 0x65, 0x66, 0x65, 0x72, 0x65, 0x6E, 0x63, 0x65, 0x64, 0x2D, 0x64, 0x61, 0x74, 0x61, + 0x2D, 0x66, 0x69, 0x6C, 0x65, 0x22, 0x3A, 0x20, 0x22, 0x2F, 0x64, 0x61, 0x74, 0x61, 0x2F, 0x74, + 0x61, 0x62, 0x6C, 0x65, 0x2F, 0x70, 0x61, 0x72, 0x74, 0x2D, 0x30, 0x30, 0x30, 0x30, 0x30, 0x2E, + 0x70, 0x61, 0x72, 0x71, 0x75, 0x65, 0x74, 0x22, 0x2C, 0x20, 0x22, 0x63, 0x61, 0x72, 0x64, 0x69, + 0x6E, 0x61, 0x6C, 0x69, 0x74, 0x79, 0x22, 0x3A, 0x20, 0x22, 0x32, 0x22, 0x7D, 0x7D, 0x5D, 0x7D, + 0x6C, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x50, 0x46, 0x41, 0x31, +}; + +} + +TEST(PuffinDeletionVectorFooterBind, AcceptsMixedBlobTypesAndBindsDeletionVector) +{ + ReadBufferFromMemory file(mixed_sketch_and_dv_puffin, sizeof(mixed_sketch_and_dv_puffin)); + const auto blobs = readPuffinFooterBlobsFromSeekable(file, sizeof(mixed_sketch_and_dv_puffin)); + ASSERT_EQ(blobs.size(), 2u); + EXPECT_EQ(blobs[0].type, "apache-datasketches-theta-v1"); + EXPECT_EQ(blobs[1].type, "deletion-vector-v1"); + + const auto & blob = bindDeletionVectorBlob( + blobs, /*content_offset=*/20, /*content_size_in_bytes=*/44, "/data/table/part-00000.parquet", 2); + EXPECT_EQ(blob.type, "deletion-vector-v1"); + + ReadBufferFromMemory payload(mixed_sketch_and_dv_puffin, sizeof(mixed_sketch_and_dv_puffin)); + const auto positions = readDeletionVectorFromPuffin(payload, 20, 44, /*expected_cardinality=*/2); + ASSERT_EQ(positions.size(), 2u); + EXPECT_EQ(positions[0], 2u); + EXPECT_EQ(positions[1], 5u); +} + +TEST(PuffinDeletionVectorFooterBind, RejectsBindToNonDeletionVectorBlob) +{ + ReadBufferFromMemory file(mixed_sketch_and_dv_puffin, sizeof(mixed_sketch_and_dv_puffin)); + const auto blobs = readPuffinFooterBlobsFromSeekable(file, sizeof(mixed_sketch_and_dv_puffin)); + try + { + bindDeletionVectorBlob(blobs, /*content_offset=*/4, /*content_size_in_bytes=*/16, "/data/table/part-00000.parquet", 2); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::BAD_ARGUMENTS); + EXPECT_NE(e.message().find("expected deletion-vector-v1"), std::string::npos); + } +} + +TEST(PuffinDeletionVectorFooterBind, RejectsColumnScopedFields) +{ + auto blobs = readFixtureBlobs(); + ASSERT_FALSE(blobs.empty()); + blobs[0].fields = {1}; + + try + { + bindDeletionVectorBlob(blobs, /*content_offset=*/4, /*content_size_in_bytes=*/44, "/data/file_a.parquet", 2); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::BAD_ARGUMENTS); + EXPECT_NE(e.message().find("unsupported non-empty 'fields'"), std::string::npos); + } +} + +TEST(PuffinDeletionVectorFooterBind, AcceptsIcebergRowPositionFieldMarker) +{ + auto blobs = readFixtureBlobs(); + ASSERT_FALSE(blobs.empty()); + /// Spark-written file-scoped DVs use fields=[_pos] (2147483645), not []. + blobs[0].fields = {ICEBERG_ROW_POSITION_FIELD_ID}; + + EXPECT_NO_THROW( + bindDeletionVectorBlob(blobs, /*content_offset=*/4, /*content_size_in_bytes=*/44, "/data/file_a.parquet", 2)); +} + +TEST(PuffinDeletionVectorFooterBind, RejectsNonMinusOneSnapshotOrSequence) +{ + auto blobs = readFixtureBlobs(); + ASSERT_FALSE(blobs.empty()); + blobs[0].snapshot_id = 0; + + try + { + bindDeletionVectorBlob(blobs, /*content_offset=*/4, /*content_size_in_bytes=*/44, "/data/file_a.parquet", 2); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::BAD_ARGUMENTS); + EXPECT_NE(e.message().find("snapshot-id and sequence-number must be -1"), std::string::npos); + } + + blobs = readFixtureBlobs(); + blobs[0].sequence_number = 0; + try + { + bindDeletionVectorBlob(blobs, /*content_offset=*/4, /*content_size_in_bytes=*/44, "/data/file_a.parquet", 2); + FAIL() << "Expected exception"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::BAD_ARGUMENTS); + EXPECT_NE(e.message().find("snapshot-id and sequence-number must be -1"), std::string::npos); + } +} diff --git a/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_dv_referenced_data_file.cpp b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_dv_referenced_data_file.cpp new file mode 100644 index 000000000000..dba4b03f5244 --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_dv_referenced_data_file.cpp @@ -0,0 +1,71 @@ +#include + +#include + +#if USE_AVRO + +#include +#include +#include + +using namespace DB; +using namespace DB::Iceberg; + +namespace DB::ErrorCodes +{ +extern const int ICEBERG_SPECIFICATION_VIOLATION; +} + +TEST(PuffinDeletionVectorReferencedDataFile, AcceptsNonEmptyDirectField) +{ + const auto path = IcebergPathFromMetadata::deserialize("/data/file.parquet"); + const auto manifest = IcebergPathFromMetadata::deserialize("/meta/manifest.avro"); + EXPECT_NO_THROW(requireDirectReferencedDataFileForPuffinDeletionVector(/*set_from_referenced_data_file_field=*/true, path, manifest)); +} + +TEST(PuffinDeletionVectorReferencedDataFile, RejectsBoundsOnlyFallback) +{ + const auto path = IcebergPathFromMetadata::deserialize("/data/file.parquet"); + const auto manifest = IcebergPathFromMetadata::deserialize("/meta/manifest.avro"); + try + { + requireDirectReferencedDataFileForPuffinDeletionVector(/*set_from_referenced_data_file_field=*/false, path, manifest); + FAIL() << "Expected ICEBERG_SPECIFICATION_VIOLATION"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION); + EXPECT_TRUE(e.message().find("referenced_data_file") != std::string::npos); + } +} + +TEST(PuffinDeletionVectorReferencedDataFile, RejectsMissingPath) +{ + const auto manifest = IcebergPathFromMetadata::deserialize("/meta/manifest.avro"); + try + { + requireDirectReferencedDataFileForPuffinDeletionVector(/*set_from_referenced_data_file_field=*/true, std::nullopt, manifest); + FAIL() << "Expected ICEBERG_SPECIFICATION_VIOLATION"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION); + } +} + +TEST(PuffinDeletionVectorReferencedDataFile, RejectsEmptyPath) +{ + const auto empty_path = IcebergPathFromMetadata::deserialize(""); + const auto manifest = IcebergPathFromMetadata::deserialize("/meta/manifest.avro"); + try + { + requireDirectReferencedDataFileForPuffinDeletionVector(/*set_from_referenced_data_file_field=*/true, empty_path, manifest); + FAIL() << "Expected ICEBERG_SPECIFICATION_VIOLATION"; + } + catch (const Exception & e) + { + EXPECT_EQ(e.code(), ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION); + } +} + +#endif diff --git a/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_files_cache_clone.cpp b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_files_cache_clone.cpp new file mode 100644 index 000000000000..584a8f76b984 --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_files_cache_clone.cpp @@ -0,0 +1,92 @@ +#include + +#include +#include +#include + +using namespace DB; + +namespace CurrentMetrics +{ +extern const Metric PuffinFilesCacheBytes; +extern const Metric PuffinFilesCacheFiles; +} + +namespace +{ + +DataLakeObjectMetadata::ExcludedRowsPtr makeExcludedRows(const std::vector & positions) +{ + auto excluded_rows = std::make_shared(); + for (size_t position : positions) + excluded_rows->add(position); + return excluded_rows; +} + +} + +TEST(PuffinFilesCacheClone, CacheHitReturnsIndependentCopy) +{ + PuffinFilesCache cache("SLRU", 1'000'000, 100, 0.5); + + const auto key = PuffinFilesCache::tryCreateKey( + "Local:////test-prefix", "puffin.bin", "etag-1", 100, 200, "data/file-a.parquet", 3, 100); + ASSERT_TRUE(key.has_value()); + + size_t load_calls = 0; + auto load_fn = [&]() + { + ++load_calls; + return makeExcludedRows({1, 5, 10}); + }; + + auto first = cache.getOrSetDeletionVector(*key, load_fn); + auto second = cache.getOrSetDeletionVector(*key, load_fn); + + ASSERT_EQ(load_calls, 1); + ASSERT_NE(first, second); + EXPECT_TRUE(first->rb_contains(1)); + EXPECT_TRUE(first->rb_contains(5)); + EXPECT_TRUE(first->rb_contains(10)); + EXPECT_FALSE(first->rb_contains(99)); + + first->add(99); + + EXPECT_FALSE(second->rb_contains(99)); + + auto third = cache.getOrSetDeletionVector(*key, load_fn); + ASSERT_NE(third, first); + EXPECT_FALSE(third->rb_contains(99)); + EXPECT_TRUE(third->rb_contains(10)); +} + +TEST(PuffinFilesCacheClone, EmptyExcludedRowsReturnsNullptr) +{ + PuffinFilesCache cache("SLRU", 1'000'000, 100, 0.5); + + const auto key = PuffinFilesCache::tryCreateKey( + "Local:////test-prefix", "puffin.bin", "etag-1", 100, 200, "data/file-a.parquet", 3, 100); + ASSERT_TRUE(key.has_value()); + + const auto files_before = CurrentMetrics::get(CurrentMetrics::PuffinFilesCacheFiles); + const auto bytes_before = CurrentMetrics::get(CurrentMetrics::PuffinFilesCacheBytes); + + size_t load_calls = 0; + auto load_fn = [&]() + { + ++load_calls; + return DataLakeObjectMetadata::ExcludedRowsPtr{}; + }; + + auto first = cache.getOrSetDeletionVector(*key, load_fn); + auto second = cache.getOrSetDeletionVector(*key, load_fn); + + const auto expected_weight = PuffinFilesCacheCell::calculateMemorySize( + /*is_empty_deletion_vector_=*/true, nullptr, key->approximateMemoryBytes()); + + EXPECT_EQ(first, nullptr); + EXPECT_EQ(second, nullptr); + EXPECT_EQ(load_calls, 1); + EXPECT_EQ(CurrentMetrics::get(CurrentMetrics::PuffinFilesCacheFiles), files_before + 1); + EXPECT_EQ(CurrentMetrics::get(CurrentMetrics::PuffinFilesCacheBytes), bytes_before + static_cast(expected_weight)); +} diff --git a/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_files_cache_key.cpp b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_files_cache_key.cpp new file mode 100644 index 000000000000..0c5352ba741d --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_files_cache_key.cpp @@ -0,0 +1,199 @@ +#include + +#include +#include +#include + +using namespace DB; + +namespace +{ + +constexpr const char * kDefaultStorageIdentity = "Local:////test-prefix"; + +std::optional makeKey( + const String & referenced_data_file, + UInt64 expected_cardinality = 2, + UInt64 data_file_record_count = 100, + const String & storage_identity = kDefaultStorageIdentity) +{ + return PuffinFilesCache::tryCreateKey( + storage_identity, + "puffin.bin", + "etag-1", + 100, + 200, + referenced_data_file, + expected_cardinality, + data_file_record_count); +} + +DataLakeObjectMetadata::ExcludedRowsPtr makeExcludedRows(const std::vector & positions) +{ + auto excluded_rows = std::make_shared(); + for (size_t position : positions) + excluded_rows->add(position); + return excluded_rows; +} + +} + +TEST(PuffinFilesCacheKey, SamePuffinSliceDifferentReferencedDataFile) +{ + const auto key1 = makeKey("data/file-a.parquet"); + const auto key2 = makeKey("data/file-b.parquet"); + + ASSERT_TRUE(key1.has_value()); + ASSERT_TRUE(key2.has_value()); + EXPECT_NE(*key1, *key2); + EXPECT_NE(PuffinFilesCacheKeyHash{}(*key1), PuffinFilesCacheKeyHash{}(*key2)); +} + +TEST(PuffinFilesCacheKey, SameReferencedDataFileProducesEqualKeys) +{ + const auto key1 = makeKey("data/file-a.parquet"); + const auto key2 = makeKey("data/file-a.parquet"); + + ASSERT_TRUE(key1.has_value()); + ASSERT_TRUE(key2.has_value()); + EXPECT_EQ(*key1, *key2); + EXPECT_EQ(PuffinFilesCacheKeyHash{}(*key1), PuffinFilesCacheKeyHash{}(*key2)); +} + +TEST(PuffinFilesCacheKey, DifferentExpectedCardinalityProducesUnequalKeys) +{ + const auto key1 = makeKey("data/file-a.parquet", /*expected_cardinality=*/2); + const auto key2 = makeKey("data/file-a.parquet", /*expected_cardinality=*/3); + + ASSERT_TRUE(key1.has_value()); + ASSERT_TRUE(key2.has_value()); + EXPECT_NE(*key1, *key2); + EXPECT_NE(PuffinFilesCacheKeyHash{}(*key1), PuffinFilesCacheKeyHash{}(*key2)); +} + +TEST(PuffinFilesCacheKey, DifferentDataFileRecordCountProducesUnequalKeys) +{ + const auto key1 = makeKey("data/file-a.parquet", /*expected_cardinality=*/2, /*data_file_record_count=*/100); + const auto key2 = makeKey("data/file-a.parquet", /*expected_cardinality=*/2, /*data_file_record_count=*/50); + + ASSERT_TRUE(key1.has_value()); + ASSERT_TRUE(key2.has_value()); + EXPECT_NE(*key1, *key2); + EXPECT_NE(PuffinFilesCacheKeyHash{}(*key1), PuffinFilesCacheKeyHash{}(*key2)); +} + +TEST(PuffinFilesCacheKey, DifferentStorageIdentityProducesUnequalKeys) +{ + const auto key1 = makeKey("data/file-a.parquet", 2, 100, "S3://http://minio-a:9001/bucket/warehouse"); + const auto key2 = makeKey("data/file-a.parquet", 2, 100, "S3://http://minio-a:9001/bucket-b/warehouse"); + + ASSERT_TRUE(key1.has_value()); + ASSERT_TRUE(key2.has_value()); + EXPECT_NE(*key1, *key2); + EXPECT_NE(PuffinFilesCacheKeyHash{}(*key1), PuffinFilesCacheKeyHash{}(*key2)); +} + +TEST(PuffinFilesCacheKey, DifferentEndpointSameBucketPrefixProducesUnequalKeys) +{ + /// Same bucket + prefix, different getDescription() (S3 endpoint). + const auto key1 = makeKey("data/file-a.parquet", 2, 100, "S3://http://minio-a:9001/bucket/warehouse"); + const auto key2 = makeKey("data/file-a.parquet", 2, 100, "S3://http://minio-b:9001/bucket/warehouse"); + + ASSERT_TRUE(key1.has_value()); + ASSERT_TRUE(key2.has_value()); + EXPECT_NE(*key1, *key2); + EXPECT_NE(PuffinFilesCacheKeyHash{}(*key1), PuffinFilesCacheKeyHash{}(*key2)); +} + +TEST(PuffinFilesCacheKey, EmptyEtagReturnsNullopt) +{ + const auto key = PuffinFilesCache::tryCreateKey( + kDefaultStorageIdentity, "puffin.bin", "", 100, 200, "data/file-a.parquet", 2, 100); + EXPECT_FALSE(key.has_value()); +} + +TEST(PuffinFilesCacheKey, DifferentStorageIdentityDoesNotHitShare) +{ + PuffinFilesCache cache("SLRU", 1'000'000, 100, 0.5); + + const auto key_a = makeKey("data/file-a.parquet", 2, 100, "S3://http://minio-a:9001/bucket/warehouse"); + const auto key_b = makeKey("data/file-a.parquet", 2, 100, "S3://http://minio-a:9001/bucket-b/warehouse"); + ASSERT_TRUE(key_a.has_value()); + ASSERT_TRUE(key_b.has_value()); + + size_t load_a_calls = 0; + size_t load_b_calls = 0; + + auto first = cache.getOrSetDeletionVector(*key_a, [&]() + { + ++load_a_calls; + return makeExcludedRows({1, 2}); + }); + + auto second = cache.getOrSetDeletionVector(*key_b, [&]() + { + ++load_b_calls; + return makeExcludedRows({10, 20}); + }); + + ASSERT_EQ(load_a_calls, 1); + ASSERT_EQ(load_b_calls, 1); + ASSERT_TRUE(first); + ASSERT_TRUE(second); + EXPECT_TRUE(first->rb_contains(1)); + EXPECT_FALSE(first->rb_contains(10)); + EXPECT_TRUE(second->rb_contains(10)); + EXPECT_FALSE(second->rb_contains(1)); + + auto third = cache.getOrSetDeletionVector(*key_b, [&]() + { + ++load_b_calls; + return makeExcludedRows({99}); + }); + + ASSERT_EQ(load_b_calls, 1); + ASSERT_TRUE(third); + EXPECT_TRUE(third->rb_contains(10)); + EXPECT_FALSE(third->rb_contains(99)); +} + +TEST(PuffinFilesCacheKey, DifferentEndpointDoesNotHitShare) +{ + PuffinFilesCache cache("SLRU", 1'000'000, 100, 0.5); + + const auto key_a = makeKey("data/file-a.parquet", 2, 100, "S3://http://minio-a:9001/bucket/warehouse"); + const auto key_b = makeKey("data/file-a.parquet", 2, 100, "S3://http://minio-b:9001/bucket/warehouse"); + ASSERT_TRUE(key_a.has_value()); + ASSERT_TRUE(key_b.has_value()); + + size_t load_a_calls = 0; + size_t load_b_calls = 0; + + auto first = cache.getOrSetDeletionVector(*key_a, [&]() + { + ++load_a_calls; + return makeExcludedRows({1, 2}); + }); + + auto second = cache.getOrSetDeletionVector(*key_b, [&]() + { + ++load_b_calls; + return makeExcludedRows({10, 20}); + }); + + ASSERT_EQ(load_a_calls, 1); + ASSERT_EQ(load_b_calls, 1); + ASSERT_TRUE(first); + ASSERT_TRUE(second); + EXPECT_TRUE(first->rb_contains(1)); + EXPECT_FALSE(first->rb_contains(10)); + EXPECT_TRUE(second->rb_contains(10)); + EXPECT_FALSE(second->rb_contains(1)); +} + +TEST(PuffinFilesCacheKey, MakeStorageIdentityIncludesDescription) +{ + LocalObjectStorage storage(LocalObjectStorageSettings("disk", "warehouse", /*read_only=*/true)); + const auto identity = PuffinFilesCache::makeStorageIdentity(storage); + EXPECT_EQ(identity, "Local://" + storage.getDescription() + "//warehouse"); +} diff --git a/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_files_cache_metrics.cpp b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_files_cache_metrics.cpp new file mode 100644 index 000000000000..91b730bf83d7 --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_files_cache_metrics.cpp @@ -0,0 +1,203 @@ +#include + +#include +#include +#include +#include + +#include +#include +#include +#include + +using namespace DB; + +namespace ProfileEvents +{ +extern const Event PuffinFilesCacheHits; +extern const Event PuffinFilesCacheMisses; +} + +namespace +{ + +DataLakeObjectMetadata::ExcludedRowsPtr makeExcludedRows(const std::vector & positions) +{ + auto excluded_rows = std::make_shared(); + for (size_t position : positions) + excluded_rows->add(position); + return excluded_rows; +} + +} + +TEST(PuffinFilesCacheMetrics, ClearDuringLoadCountsAsMissNotHit) +{ + PuffinFilesCache cache("SLRU", 1'000'000, 100, 0.5); + + const auto key = PuffinFilesCache::tryCreateKey( + "Local:////test-prefix", "puffin.bin", "etag-1", 100, 200, "data/file-a.parquet", 1, 100); + ASSERT_TRUE(key.has_value()); + + auto & counters = CurrentThread::getProfileEvents(); + const auto hits_before = counters[ProfileEvents::PuffinFilesCacheHits].load(); + const auto misses_before = counters[ProfileEvents::PuffinFilesCacheMisses].load(); + + size_t load_calls = 0; + const auto result = cache.getOrSetDeletionVector( + *key, + [&]() + { + ++load_calls; + /// Simulate concurrent SYSTEM DROP while this key is loading: CacheBase then returns + /// `{value, false}` because the insert token was discarded. + cache.clear(); + return makeExcludedRows({1}); + }); + + ASSERT_EQ(load_calls, 1u); + ASSERT_NE(result, nullptr); + EXPECT_TRUE(result->rb_contains(1)); + EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheHits].load() - hits_before, 0u); + EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheMisses].load() - misses_before, 1u); + + /// Entry was not inserted after clear; the next lookup must load again. + const auto second = cache.getOrSetDeletionVector(*key, [&]() + { + ++load_calls; + return makeExcludedRows({1}); + }); + ASSERT_EQ(load_calls, 2u); + ASSERT_NE(second, nullptr); + EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheMisses].load() - misses_before, 2u); + EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheHits].load() - hits_before, 0u); +} + +TEST(PuffinFilesCacheMetrics, WaiterOfClearDiscardedLoadCountsAsMiss) +{ + PuffinFilesCache cache("SLRU", 1'000'000, 100, 0.5); + + const auto key = PuffinFilesCache::tryCreateKey( + "Local:////test-prefix", "puffin.bin", "etag-waiter", 100, 200, "data/file-w.parquet", 1, 100); + ASSERT_TRUE(key.has_value()); + + const auto hits_before = ProfileEvents::global_counters[ProfileEvents::PuffinFilesCacheHits].load(); + const auto misses_before = ProfileEvents::global_counters[ProfileEvents::PuffinFilesCacheMisses].load(); + + std::promise load_started; + auto load_started_future = load_started.get_future(); + + std::atomic load_calls{0}; + std::atomic waiter_load_called{false}; + std::atomic waiter_joined_insert_token{false}; + + std::thread producer( + [&]() + { + cache.getOrSetDeletionVector( + *key, + [&]() + { + ++load_calls; + load_started.set_value(); + + /// Wait until the waiter has acquired the same insert token (refcount >= 2) + /// before clear()+finish. A fixed sleep raced: if the producer finished first, + /// the waiter started a fresh load and this test became flaky. + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(5); + while (cache.getInsertTokenRefcount(*key) < 2) + { + if (std::chrono::steady_clock::now() >= deadline) + return makeExcludedRows({42}); + std::this_thread::yield(); + } + waiter_joined_insert_token.store(true); + + cache.clear(); + return makeExcludedRows({42}); + }); + }); + + ASSERT_EQ(load_started_future.wait_for(std::chrono::seconds(5)), std::future_status::ready); + + std::thread waiter( + [&]() + { + cache.getOrSetDeletionVector( + *key, + [&]() + { + waiter_load_called.store(true); + return makeExcludedRows({99}); + }); + }); + + producer.join(); + waiter.join(); + + ASSERT_TRUE(waiter_joined_insert_token.load()); + EXPECT_EQ(load_calls.load(), 1u); + EXPECT_FALSE(waiter_load_called.load()); + EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::PuffinFilesCacheHits] - hits_before, 0u); + EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::PuffinFilesCacheMisses] - misses_before, 2u); +} + +TEST(PuffinFilesCacheMetrics, OrdinaryHitAndMissCounters) +{ + PuffinFilesCache cache("SLRU", 1'000'000, 100, 0.5); + + const auto key = PuffinFilesCache::tryCreateKey( + "Local:////test-prefix", "puffin.bin", "etag-2", 100, 200, "data/file-b.parquet", 1, 100); + ASSERT_TRUE(key.has_value()); + + auto & counters = CurrentThread::getProfileEvents(); + const auto hits_before = counters[ProfileEvents::PuffinFilesCacheHits].load(); + const auto misses_before = counters[ProfileEvents::PuffinFilesCacheMisses].load(); + + size_t load_calls = 0; + auto load_fn = [&]() + { + ++load_calls; + return makeExcludedRows({7}); + }; + + ASSERT_NE(cache.getOrSetDeletionVector(*key, load_fn), nullptr); + ASSERT_NE(cache.getOrSetDeletionVector(*key, load_fn), nullptr); + + EXPECT_EQ(load_calls, 1u); + EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheMisses].load() - misses_before, 1u); + EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheHits].load() - hits_before, 1u); +} + +TEST(PuffinFilesCacheMetrics, HitRemainsHitWhenCacheClearedAfterLookup) +{ + PuffinFilesCache cache("SLRU", 1'000'000, 100, 0.5); + + const auto key = PuffinFilesCache::tryCreateKey( + "Local:////test-prefix", "puffin.bin", "etag-hit-clear", 100, 200, "data/file-c.parquet", 1, 100); + ASSERT_TRUE(key.has_value()); + + auto & counters = CurrentThread::getProfileEvents(); + size_t load_calls = 0; + auto load_fn = [&]() + { + ++load_calls; + return makeExcludedRows({3}); + }; + + ASSERT_NE(cache.getOrSetDeletionVector(*key, load_fn), nullptr); + + const auto hits_before = counters[ProfileEvents::PuffinFilesCacheHits].load(); + const auto misses_before = counters[ProfileEvents::PuffinFilesCacheMisses].load(); + + ASSERT_NE(cache.getOrSetDeletionVector(*key, load_fn), nullptr); + EXPECT_EQ(load_calls, 1u); + EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheHits].load() - hits_before, 1u); + EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheMisses].load() - misses_before, 0u); + + /// Clearing after the hit must not rewrite the already-recorded hit as a miss. The old + /// contains()-after-getOrSet path could race here with SYSTEM DROP PUFFIN FILES CACHE. + cache.clear(); + EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheHits].load() - hits_before, 1u); + EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheMisses].load() - misses_before, 0u); +} diff --git a/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_files_cache_weight.cpp b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_files_cache_weight.cpp new file mode 100644 index 000000000000..8d7db1e39ac6 --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_files_cache_weight.cpp @@ -0,0 +1,142 @@ +#include + +#include +#include +#include +#include +#include + +using namespace DB; + +namespace CurrentMetrics +{ +extern const Metric PuffinFilesCacheBytes; +extern const Metric PuffinFilesCacheFiles; +} + +namespace +{ + +DataLakeObjectMetadata::ExcludedRowsPtr makeLargeSparseExcludedRows(size_t keys = 33) +{ + auto excluded_rows = std::make_shared(); + for (size_t i = 0; i < keys; ++i) + excluded_rows->add((static_cast(i) + 1) << 32); + return excluded_rows; +} + +roaring::Roaring64Map makeLargeSparseRoaring64(size_t keys = 33) +{ + roaring::Roaring64Map bitmap; + for (size_t i = 0; i < keys; ++i) + bitmap.add((static_cast(i) + 1) << 32); + return bitmap; +} + +PuffinFilesCacheKey makeUniqueKey(size_t index, const String & long_suffix = "") +{ + auto key = PuffinFilesCache::tryCreateKey( + "Local:////test-prefix", + "puffin.bin", + "etag-" + std::to_string(index) + long_suffix, + 100, + 200, + "data/file-" + std::to_string(index) + ".parquet" + long_suffix, + /*expected_cardinality=*/0, + /*data_file_record_count=*/1000); + EXPECT_TRUE(key.has_value()); + return *key; +} + +} + +TEST(RoaringBitmapWithSmallSetMemory, LargeSparseBitmapAllocatedBytesExceedCardinalityEstimate) +{ + const auto excluded_rows = makeLargeSparseExcludedRows(); + + ASSERT_TRUE(excluded_rows->isLarge()); + EXPECT_GT(excluded_rows->getAllocatedBytes(), excluded_rows->size() * sizeof(size_t)); +} + +TEST(RoaringBitmapWithSmallSetMemory, LargeSparseBitmapAllocatedBytesExceedSerializedSize) +{ + const auto excluded_rows = makeLargeSparseExcludedRows(); + const auto serialized = makeLargeSparseRoaring64().getSizeInBytes(/*portable=*/true); + + ASSERT_TRUE(excluded_rows->isLarge()); + EXPECT_GT(excluded_rows->getAllocatedBytes(), serialized); +} + +TEST(PuffinFilesCacheWeight, UsesRoaringAllocatedBytesForWeight) +{ + const auto excluded_rows = makeLargeSparseExcludedRows(); + const auto key = PuffinFilesCache::tryCreateKey( + "Local:////test-prefix", "puffin.bin", "etag-1", 100, 200, "data/file-a.parquet", 33, 1000); + ASSERT_TRUE(key.has_value()); + + PuffinFilesCache cache("SLRU", 1'000'000, 100, 0.5); + cache.getOrSetDeletionVector(*key, [&]() { return excluded_rows; }); + + EXPECT_GE(CurrentMetrics::get(CurrentMetrics::PuffinFilesCacheBytes), excluded_rows->getAllocatedBytes()); +} + +TEST(PuffinFilesCacheWeight, EvictsSparseBitmapsUnderSmallByteLimit) +{ + /// Each sparse entry weighs far more than cardinality × sizeof(size_t); a tiny byte + /// limit must not retain many of them if weight uses allocated bytes. + const auto sample_key = makeUniqueKey(0); + const auto sample_rows = makeLargeSparseExcludedRows(16); + const auto entry_weight = PuffinFilesCacheCell::calculateMemorySize( + /*is_empty_deletion_vector_=*/false, sample_rows, sample_key.approximateMemoryBytes()); + ASSERT_GT(entry_weight, 0u); + + const size_t max_bytes = static_cast(entry_weight) + 64; // roughly one entry + PuffinFilesCache cache("SLRU", max_bytes, /*max_count=*/100, /*size_ratio=*/0.5); + + for (size_t i = 0; i < 8; ++i) + { + const auto key = makeUniqueKey(i); + cache.getOrSetDeletionVector(key, [&]() { return makeLargeSparseExcludedRows(16); }); + } + + EXPECT_LE(CurrentMetrics::get(CurrentMetrics::PuffinFilesCacheBytes), static_cast(max_bytes)); + EXPECT_LT(CurrentMetrics::get(CurrentMetrics::PuffinFilesCacheFiles), 8); +} + +TEST(PuffinFilesCacheWeight, LongKeyEmptyEntriesEvictAtByteLimit) +{ + /// Empty DVs used to weigh 1 byte and ignore key strings. With an unlimited entry + /// count, long unique keys must still be bounded by the configured byte limit. + const String long_suffix(8 * 1024, 'x'); + const auto sample_key = makeUniqueKey(0, long_suffix); + const auto entry_weight = PuffinFilesCacheCell::calculateMemorySize( + /*is_empty_deletion_vector_=*/true, nullptr, sample_key.approximateMemoryBytes()); + ASSERT_GT(entry_weight, sample_key.approximateMemoryBytes()); + ASSERT_GT(entry_weight, 8 * 1024u); + + /// Allow roughly two long-key empty entries; inserting many more must evict. + const size_t max_bytes = static_cast(entry_weight) * 2 + 128; + PuffinFilesCache cache("SLRU", max_bytes, /*max_count=*/0, /*size_ratio=*/0.5); + + for (size_t i = 0; i < 32; ++i) + { + const auto key = makeUniqueKey(i, long_suffix); + cache.getOrSetDeletionVector(key, []() -> DataLakeObjectMetadata::ExcludedRowsPtr { return nullptr; }); + } + + EXPECT_LE(CurrentMetrics::get(CurrentMetrics::PuffinFilesCacheBytes), static_cast(max_bytes)); + EXPECT_LE(CurrentMetrics::get(CurrentMetrics::PuffinFilesCacheFiles), 3); +} + +TEST(PuffinFilesCacheWeight, EmptyEntryChargesKeyNotOneByte) +{ + const String long_path(4096, 'p'); + const auto key = PuffinFilesCache::tryCreateKey( + "Local:////test-prefix", long_path, "etag-empty", 0, 0, long_path, 0, 0); + ASSERT_TRUE(key.has_value()); + + const auto weight = PuffinFilesCacheCell::calculateMemorySize( + /*is_empty_deletion_vector_=*/true, nullptr, key->approximateMemoryBytes()); + EXPECT_GT(weight, key->approximateMemoryBytes()); + EXPECT_GE(weight, 2 * 4096u); +} diff --git a/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_footer_cache.cpp b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_footer_cache.cpp new file mode 100644 index 000000000000..a7c492e9e391 --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_footer_cache.cpp @@ -0,0 +1,247 @@ +#include + +#include +#include +#include +#include +#include + +using namespace DB; + +namespace ProfileEvents +{ +extern const Event PuffinFilesRead; +} + +namespace +{ + +/// Two equal-cardinality deletion-vector-v1 blobs for different data files. +constexpr UInt8 two_equal_cardinality_dvs_puffin[] = { + 0x50, 0x46, 0x41, 0x31, 0x00, 0x00, 0x00, 0x24, 0xD1, 0xD3, 0x39, 0x64, 0x01, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x3A, 0x30, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x01, 0x00, 0x10, 0x00, 0x00, 0x00, 0x02, 0x00, 0x05, 0x00, 0x2C, 0xDB, 0x9F, 0xC1, + 0x00, 0x00, 0x00, 0x24, 0xD1, 0xD3, 0x39, 0x64, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x3A, 0x30, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, + 0x10, 0x00, 0x00, 0x00, 0x07, 0x00, 0x09, 0x00, 0xB7, 0xB0, 0x20, 0xFF, 0x50, 0x46, 0x41, 0x31, + 0x7B, 0x22, 0x62, 0x6C, 0x6F, 0x62, 0x73, 0x22, 0x3A, 0x20, 0x5B, 0x7B, 0x22, 0x74, 0x79, 0x70, + 0x65, 0x22, 0x3A, 0x20, 0x22, 0x64, 0x65, 0x6C, 0x65, 0x74, 0x69, 0x6F, 0x6E, 0x2D, 0x76, 0x65, + 0x63, 0x74, 0x6F, 0x72, 0x2D, 0x76, 0x31, 0x22, 0x2C, 0x20, 0x22, 0x66, 0x69, 0x65, 0x6C, 0x64, + 0x73, 0x22, 0x3A, 0x20, 0x5B, 0x5D, 0x2C, 0x20, 0x22, 0x73, 0x6E, 0x61, 0x70, 0x73, 0x68, 0x6F, + 0x74, 0x2D, 0x69, 0x64, 0x22, 0x3A, 0x20, 0x2D, 0x31, 0x2C, 0x20, 0x22, 0x73, 0x65, 0x71, 0x75, + 0x65, 0x6E, 0x63, 0x65, 0x2D, 0x6E, 0x75, 0x6D, 0x62, 0x65, 0x72, 0x22, 0x3A, 0x20, 0x2D, 0x31, + 0x2C, 0x20, 0x22, 0x6F, 0x66, 0x66, 0x73, 0x65, 0x74, 0x22, 0x3A, 0x20, 0x34, 0x2C, 0x20, 0x22, + 0x6C, 0x65, 0x6E, 0x67, 0x74, 0x68, 0x22, 0x3A, 0x20, 0x34, 0x34, 0x2C, 0x20, 0x22, 0x70, 0x72, + 0x6F, 0x70, 0x65, 0x72, 0x74, 0x69, 0x65, 0x73, 0x22, 0x3A, 0x20, 0x7B, 0x22, 0x72, 0x65, 0x66, + 0x65, 0x72, 0x65, 0x6E, 0x63, 0x65, 0x64, 0x2D, 0x64, 0x61, 0x74, 0x61, 0x2D, 0x66, 0x69, 0x6C, + 0x65, 0x22, 0x3A, 0x20, 0x22, 0x2F, 0x64, 0x61, 0x74, 0x61, 0x2F, 0x66, 0x69, 0x6C, 0x65, 0x5F, + 0x61, 0x2E, 0x70, 0x61, 0x72, 0x71, 0x75, 0x65, 0x74, 0x22, 0x2C, 0x20, 0x22, 0x63, 0x61, 0x72, + 0x64, 0x69, 0x6E, 0x61, 0x6C, 0x69, 0x74, 0x79, 0x22, 0x3A, 0x20, 0x22, 0x32, 0x22, 0x7D, 0x7D, + 0x2C, 0x20, 0x7B, 0x22, 0x74, 0x79, 0x70, 0x65, 0x22, 0x3A, 0x20, 0x22, 0x64, 0x65, 0x6C, 0x65, + 0x74, 0x69, 0x6F, 0x6E, 0x2D, 0x76, 0x65, 0x63, 0x74, 0x6F, 0x72, 0x2D, 0x76, 0x31, 0x22, 0x2C, + 0x20, 0x22, 0x66, 0x69, 0x65, 0x6C, 0x64, 0x73, 0x22, 0x3A, 0x20, 0x5B, 0x5D, 0x2C, 0x20, 0x22, + 0x73, 0x6E, 0x61, 0x70, 0x73, 0x68, 0x6F, 0x74, 0x2D, 0x69, 0x64, 0x22, 0x3A, 0x20, 0x2D, 0x31, + 0x2C, 0x20, 0x22, 0x73, 0x65, 0x71, 0x75, 0x65, 0x6E, 0x63, 0x65, 0x2D, 0x6E, 0x75, 0x6D, 0x62, + 0x65, 0x72, 0x22, 0x3A, 0x20, 0x2D, 0x31, 0x2C, 0x20, 0x22, 0x6F, 0x66, 0x66, 0x73, 0x65, 0x74, + 0x22, 0x3A, 0x20, 0x34, 0x38, 0x2C, 0x20, 0x22, 0x6C, 0x65, 0x6E, 0x67, 0x74, 0x68, 0x22, 0x3A, + 0x20, 0x34, 0x34, 0x2C, 0x20, 0x22, 0x70, 0x72, 0x6F, 0x70, 0x65, 0x72, 0x74, 0x69, 0x65, 0x73, + 0x22, 0x3A, 0x20, 0x7B, 0x22, 0x72, 0x65, 0x66, 0x65, 0x72, 0x65, 0x6E, 0x63, 0x65, 0x64, 0x2D, + 0x64, 0x61, 0x74, 0x61, 0x2D, 0x66, 0x69, 0x6C, 0x65, 0x22, 0x3A, 0x20, 0x22, 0x2F, 0x64, 0x61, + 0x74, 0x61, 0x2F, 0x66, 0x69, 0x6C, 0x65, 0x5F, 0x62, 0x2E, 0x70, 0x61, 0x72, 0x71, 0x75, 0x65, + 0x74, 0x22, 0x2C, 0x20, 0x22, 0x63, 0x61, 0x72, 0x64, 0x69, 0x6E, 0x61, 0x6C, 0x69, 0x74, 0x79, + 0x22, 0x3A, 0x20, 0x22, 0x32, 0x22, 0x7D, 0x7D, 0x5D, 0x7D, 0x9A, 0x01, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x50, 0x46, 0x41, 0x31, +}; + +PuffinFilesCache::FooterBlobsPtr loadFixtureFooter() +{ + ReadBufferFromMemory file(two_equal_cardinality_dvs_puffin, sizeof(two_equal_cardinality_dvs_puffin)); + return std::make_shared>( + readPuffinFooterBlobsFromSeekable(file, sizeof(two_equal_cardinality_dvs_puffin))); +} + +} + +TEST(PuffinFooterMemo, CoalescedSlicesShareOneFooterParse) +{ + PuffinFilesCache cache("SLRU", 1'000'000, 100, 0.5); + + const auto footer_key = PuffinFilesCache::tryCreateFooterKey("Local:////test", "coalesced.puffin", "etag-1"); + ASSERT_TRUE(footer_key.has_value()); + + const auto key_a = PuffinFilesCache::tryCreateKey( + "Local:////test", "coalesced.puffin", "etag-1", 4, 44, "/data/file_a.parquet", 2, 100); + const auto key_b = PuffinFilesCache::tryCreateKey( + "Local:////test", "coalesced.puffin", "etag-1", 48, 44, "/data/file_b.parquet", 2, 100); + ASSERT_TRUE(key_a.has_value()); + ASSERT_TRUE(key_b.has_value()); + + auto & counters = ProfileEvents::global_counters; + const auto files_read_before = counters[ProfileEvents::PuffinFilesRead].load(); + + size_t footer_loads = 0; + auto load_footer = [&]() + { + ++footer_loads; + return loadFixtureFooter(); + }; + + /// Cold scan of two DV slices from one coalesced Puffin: one footer parse, two bitmap loads. + cache.getOrSetDeletionVector(*key_a, [&]() + { + auto footer = cache.getOrSetFooter(*footer_key, load_footer); + EXPECT_EQ(footer->size(), 2u); + bindDeletionVectorBlob(*footer, 4, 44, "/data/file_a.parquet", 2); + ReadBufferFromMemory file(two_equal_cardinality_dvs_puffin, sizeof(two_equal_cardinality_dvs_puffin)); + const auto positions = readDeletionVectorFromPuffin(file, 4, 44, 2); + auto excluded = std::make_shared(); + for (UInt64 position : positions) + excluded->add(static_cast(position)); + return excluded; + }); + + cache.getOrSetDeletionVector(*key_b, [&]() + { + auto footer = cache.getOrSetFooter(*footer_key, load_footer); + EXPECT_EQ(footer->size(), 2u); + bindDeletionVectorBlob(*footer, 48, 44, "/data/file_b.parquet", 2); + ReadBufferFromMemory file(two_equal_cardinality_dvs_puffin, sizeof(two_equal_cardinality_dvs_puffin)); + const auto positions = readDeletionVectorFromPuffin(file, 48, 44, 2); + auto excluded = std::make_shared(); + for (UInt64 position : positions) + excluded->add(static_cast(position)); + return excluded; + }); + + EXPECT_EQ(footer_loads, 1u); + /// One footer parse (`PuffinFilesRead` in readPuffinFooter) plus two blob reads. + EXPECT_EQ(counters[ProfileEvents::PuffinFilesRead].load() - files_read_before, 3u); +} + +TEST(PuffinFooterMemo, ClearDropsFooterEntries) +{ + PuffinFilesCache cache("SLRU", 1'000'000, 100, 0.5); + const auto footer_key = PuffinFilesCache::tryCreateFooterKey("Local:////test", "coalesced.puffin", "etag-1"); + ASSERT_TRUE(footer_key.has_value()); + + size_t footer_loads = 0; + auto load_footer = [&]() + { + ++footer_loads; + return loadFixtureFooter(); + }; + + ASSERT_NE(cache.getOrSetFooter(*footer_key, load_footer), nullptr); + cache.clear(); + ASSERT_NE(cache.getOrSetFooter(*footer_key, load_footer), nullptr); + EXPECT_EQ(footer_loads, 2u); +} + +TEST(PuffinFooterMemo, SizeZeroDoesNotInsert) +{ + PuffinFilesCache cache("SLRU", /*max_size_in_bytes=*/0, 100, 0.5); + const auto footer_key = PuffinFilesCache::tryCreateFooterKey("Local:////test", "coalesced.puffin", "etag-1"); + ASSERT_TRUE(footer_key.has_value()); + + size_t footer_loads = 0; + auto load_footer = [&]() + { + ++footer_loads; + return loadFixtureFooter(); + }; + + ASSERT_NE(cache.getOrSetFooter(*footer_key, load_footer), nullptr); + ASSERT_NE(cache.getOrSetFooter(*footer_key, load_footer), nullptr); + EXPECT_EQ(footer_loads, 2u); + EXPECT_EQ(cache.footerMemoEntries(), 0u); + EXPECT_EQ(cache.footerMemoBytes(), 0u); +} + +TEST(PuffinFooterMemo, SizeZeroClearsMemo) +{ + PuffinFilesCache cache("SLRU", 1'000'000, 100, 0.5); + const auto footer_key = PuffinFilesCache::tryCreateFooterKey("Local:////test", "coalesced.puffin", "etag-1"); + ASSERT_TRUE(footer_key.has_value()); + + size_t footer_loads = 0; + auto load_footer = [&]() + { + ++footer_loads; + return loadFixtureFooter(); + }; + + ASSERT_NE(cache.getOrSetFooter(*footer_key, load_footer), nullptr); + EXPECT_EQ(footer_loads, 1u); + EXPECT_EQ(cache.footerMemoEntries(), 1u); + EXPECT_GT(cache.footerMemoBytes(), 0u); + + cache.setMaxSizeInBytes(0); + EXPECT_EQ(cache.footerMemoEntries(), 0u); + EXPECT_EQ(cache.footerMemoBytes(), 0u); + + ASSERT_NE(cache.getOrSetFooter(*footer_key, load_footer), nullptr); + EXPECT_EQ(footer_loads, 2u); + EXPECT_EQ(cache.footerMemoEntries(), 0u); + + cache.setMaxSizeInBytes(1'000'000); + ASSERT_NE(cache.getOrSetFooter(*footer_key, load_footer), nullptr); + ASSERT_NE(cache.getOrSetFooter(*footer_key, load_footer), nullptr); + EXPECT_EQ(footer_loads, 3u); + EXPECT_EQ(cache.footerMemoEntries(), 1u); +} + +TEST(PuffinFooterMemo, ByteBudgetEvictsOnShrink) +{ + PuffinFilesCache cache("SLRU", 1'000'000, 100, 0.5); + const auto footer_key = PuffinFilesCache::tryCreateFooterKey("Local:////test", "coalesced.puffin", "etag-1"); + ASSERT_TRUE(footer_key.has_value()); + + size_t footer_loads = 0; + auto load_footer = [&]() + { + ++footer_loads; + return loadFixtureFooter(); + }; + + ASSERT_NE(cache.getOrSetFooter(*footer_key, load_footer), nullptr); + const UInt64 memo_bytes = cache.footerMemoBytes(); + ASSERT_GT(memo_bytes, 1u); + + /// Shrinking below current memo weight must drop retained footers. + cache.setMaxSizeInBytes(1); + EXPECT_EQ(cache.footerMemoEntries(), 0u); + EXPECT_EQ(cache.footerMemoBytes(), 0u); + + ASSERT_NE(cache.getOrSetFooter(*footer_key, load_footer), nullptr); + EXPECT_EQ(footer_loads, 2u); + /// Single fixture footer exceeds a 1-byte budget, so it must not be re-inserted. + EXPECT_EQ(cache.footerMemoEntries(), 0u); +} + +TEST(PuffinFooterMemo, CountLimitEvictsOneEntryNotAll) +{ + PuffinFilesCache cache("SLRU", 1'000'000, /*max_count=*/1, 0.5); + const auto key_a = PuffinFilesCache::tryCreateFooterKey("Local:////test", "a.puffin", "etag-a"); + const auto key_b = PuffinFilesCache::tryCreateFooterKey("Local:////test", "b.puffin", "etag-b"); + ASSERT_TRUE(key_a.has_value()); + ASSERT_TRUE(key_b.has_value()); + + size_t footer_loads = 0; + auto load_footer = [&]() + { + ++footer_loads; + return loadFixtureFooter(); + }; + + ASSERT_NE(cache.getOrSetFooter(*key_a, load_footer), nullptr); + EXPECT_EQ(cache.footerMemoEntries(), 1u); + + ASSERT_NE(cache.getOrSetFooter(*key_b, load_footer), nullptr); + /// Evict one victim for room — do not wipe the memo to empty before insert. + EXPECT_EQ(cache.footerMemoEntries(), 1u); + EXPECT_EQ(footer_loads, 2u); + + /// The retained entry must be key_b (key_a was the only victim). + ASSERT_NE(cache.getOrSetFooter(*key_b, load_footer), nullptr); + EXPECT_EQ(footer_loads, 2u); +} diff --git a/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_non_seekable_buffer_limit.cpp b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_non_seekable_buffer_limit.cpp new file mode 100644 index 000000000000..643613e6232f --- /dev/null +++ b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_non_seekable_buffer_limit.cpp @@ -0,0 +1,53 @@ +#include + +#include + +#include +#include + +#include + +using namespace DB; + +TEST(PuffinNonSeekableBufferLimit, StopsBeforeExceedingAbsoluteLimit) +{ + constexpr size_t max_buffered_size = 100; + std::vector out = {'P', 'F', 'A', '1'}; + + /// Stream after the already-buffered header would push the total over the limit. + const String rest(max_buffered_size, 'x'); + ReadBufferFromString buf(rest); + + EXPECT_THROW(appendReadBufferWithAbsoluteSizeLimit(buf, out, max_buffered_size), Exception); + EXPECT_LE(out.size(), max_buffered_size); +} + +TEST(PuffinNonSeekableBufferLimit, AcceptsInputWithinLimit) +{ + constexpr size_t max_buffered_size = 100; + std::vector out = {'P', 'F', 'A', '1'}; + + const String rest(50, 'y'); + ReadBufferFromString buf(rest); + + ASSERT_NO_THROW(appendReadBufferWithAbsoluteSizeLimit(buf, out, max_buffered_size)); + EXPECT_EQ(out.size(), 54u); + EXPECT_EQ(out[0], 'P'); + EXPECT_EQ(out[4], 'y'); +} + +TEST(PuffinNonSeekableBufferLimit, RejectsAlreadyOversizedPrefix) +{ + std::vector out(101, 'z'); + ReadBufferFromString buf(String{}); + + EXPECT_THROW(appendReadBufferWithAbsoluteSizeLimit(buf, out, /*max_buffered_size=*/100), Exception); +} + +TEST(PuffinNonSeekableBufferLimit, ProductionCeilingCoversMaxDvPlusFooter) +{ + EXPECT_EQ( + PUFFIN_NON_SEEKABLE_MAX_BUFFERED_SIZE, + PUFFIN_MAGIC_SIZE + PUFFIN_DV_MAX_BLOB_SIZE + PUFFIN_MAGIC_SIZE + PUFFIN_FOOTER_MAX_PAYLOAD_SIZE + + PUFFIN_FOOTER_TRAILER_SIZE); +} diff --git a/src/Storages/ObjectStorage/IObjectIterator.cpp b/src/Storages/ObjectStorage/IObjectIterator.cpp index 7ae3d1cbb1b2..b03ca71b325c 100644 --- a/src/Storages/ObjectStorage/IObjectIterator.cpp +++ b/src/Storages/ObjectStorage/IObjectIterator.cpp @@ -170,19 +170,19 @@ ObjectInfoPtr ObjectIteratorSplitByBuckets::next(size_t id) auto file_bucket_infos = splitter->splitToBuckets(bucket_size, *buffer, format_settings); for (const auto & file_bucket : file_bucket_infos) { - auto copy_object_info = *last_object_info; + auto copy_object_info = last_object_info->clone(); if (has_cache_entry) { auto filtered = file_bucket->filterByMatchingRowGroups(matching_row_groups); if (!filtered) continue; - copy_object_info.file_bucket_info = std::move(filtered); + copy_object_info->file_bucket_info = std::move(filtered); } else { - copy_object_info.file_bucket_info = file_bucket; + copy_object_info->file_bucket_info = file_bucket; } - pending_objects_info.push(std::make_shared(copy_object_info)); + pending_objects_info.push(std::move(copy_object_info)); } } } diff --git a/src/Storages/ObjectStorage/IObjectIterator.h b/src/Storages/ObjectStorage/IObjectIterator.h index 47febf269ee4..c922f0a70243 100644 --- a/src/Storages/ObjectStorage/IObjectIterator.h +++ b/src/Storages/ObjectStorage/IObjectIterator.h @@ -53,6 +53,10 @@ struct ObjectInfo FileBucketInfoPtr file_bucket_info; + /// Polymorphic copy. Used when splitting files into buckets so derived metadata + /// (e.g. Iceberg equality / position deletes) is not sliced away. + virtual ObjectInfoPtr clone() const { return std::make_shared(*this); } + String getIdentifier() const; }; diff --git a/src/Storages/ObjectStorage/StorageObjectStorageSource.cpp b/src/Storages/ObjectStorage/StorageObjectStorageSource.cpp index a7fd2c7000fb..a1df2e8f7f6c 100644 --- a/src/Storages/ObjectStorage/StorageObjectStorageSource.cpp +++ b/src/Storages/ObjectStorage/StorageObjectStorageSource.cpp @@ -127,6 +127,21 @@ static void logIcebergFileStats(const ObjectInfoPtr & object_info, const LoggerP #endif } +/// Count-from-files cache key is data-file identity only. Skip when row filtering can change +/// independently (DVs / selection vectors, Iceberg eq/pos deletes) or when the task is a bucket subset. +/// Cache must stay fail-closed even when need_only_count is allowed for position deletes / DVs: +/// the key is path + mtime only, and deletes change the contributed row count without touching the file. +static bool canUseCountFromFilesCache(const ObjectInfoPtr & object_info) +{ + if (hasNonEmptyExcludedRows(object_info->data_lake_metadata) || object_info->file_bucket_info) + return false; +#if USE_AVRO + if (hasIcebergEqualityDeletes(object_info) || hasIcebergPositionDeletes(object_info)) + return false; +#endif + return true; +} + StorageObjectStorageSource::StorageObjectStorageSource( const StorageID & storage_id_, String name_, @@ -633,8 +648,11 @@ Chunk StorageObjectStorageSource::generate() } } + /// Do not cache filtered cardinality: filter DAG, PREWHERE, and row policies all + /// reduce rows seen by generate(), while the cache key is file identity only. if (reader.getInputFormat() && read_context->getSettingsRef()[Setting::use_cache_for_count_from_files] - && !format_filter_info->filter_actions_dag) + && format_filter_info && !format_filter_info->hasFilter() + && canUseCountFromFilesCache(reader.getObjectInfo())) addNumRowsToCache(*reader.getObjectInfo(), total_rows_in_file); total_rows_in_file = 0; @@ -963,8 +981,23 @@ StorageObjectStorageSource::ReaderHolder StorageObjectStorageSource::createReade } } - std::optional num_rows_from_cache - = need_only_count && context_->getSettingsRef()[Setting::use_cache_for_count_from_files] ? try_get_num_rows_from_cache() : std::nullopt; + /// Equality-delete FilterTransform evaluates predicates against column values, but need_only_count + /// emits default-filled chunks — so disable the fast path for equality deletes only. + /// Position deletes and deletion vectors filter by row index (preserved on synthetic chunks); + /// DeletionVectorTransform adjusts const count chunks via roaring range cardinality, and Parquet + /// needOnlyCount reads footer/row-group metadata only (including bucketed reads). Count-from-files + /// cache stays separately fail-closed in canUseCountFromFilesCache. +#if USE_AVRO + const bool effective_need_only_count = need_only_count && !hasIcebergEqualityDeletes(object_info); +#else + const bool effective_need_only_count = need_only_count; +#endif + + const bool can_use_count_cache = effective_need_only_count + && context_->getSettingsRef()[Setting::use_cache_for_count_from_files] + && canUseCountFromFilesCache(object_info); + + std::optional num_rows_from_cache = can_use_count_cache ? try_get_num_rows_from_cache() : std::nullopt; if (num_rows_from_cache) { @@ -1137,7 +1170,7 @@ StorageObjectStorageSource::ReaderHolder StorageObjectStorageSource::createReade filter_info, true /* is_remote_fs */, compression_method, - need_only_count, + effective_need_only_count, std::nullopt /*min_block_size_bytes*/, std::nullopt /*min_block_size_rows*/, std::nullopt /*max_block_size_bytes*/); @@ -1155,22 +1188,22 @@ StorageObjectStorageSource::ReaderHolder StorageObjectStorageSource::createReade filter_info, true /* is_remote_fs */, compression_method, - need_only_count); + effective_need_only_count); } input_format->setBucketsToRead(object_info->file_bucket_info); input_format->setSerializationHints(read_from_format_info.serialization_hints); - if (need_only_count) + if (effective_need_only_count) input_format->needOnlyCount(); builder.init(Pipe(input_format)); - configuration->addDeleteTransformers(object_info, builder, format_settings, parser_shared_resources, context_); - - if (object_info->data_lake_metadata - && object_info->data_lake_metadata->excluded_rows - && object_info->data_lake_metadata->excluded_rows->size() > 0) + /// Deletion vectors (and selection vectors) address absolute file row numbers via + /// `ChunkInfoRowNumbers`. Iceberg equality deletes use a plain `FilterTransform` that + /// shrinks the chunk without maintaining `applied_filter`, so DV must run first — + /// otherwise later DV filtering maps dense post-equality indices to the wrong file rows. + if (hasNonEmptyExcludedRows(object_info->data_lake_metadata)) { builder.addSimpleTransform([&](const SharedHeader & header) { @@ -1178,6 +1211,8 @@ StorageObjectStorageSource::ReaderHolder StorageObjectStorageSource::createReade }); } + configuration->addDeleteTransformers(object_info, builder, format_settings, parser_shared_resources, context_); + std::optional schema_transform; if (object_info->data_lake_metadata && object_info->data_lake_metadata->schema_transform) { diff --git a/src/Storages/ObjectStorage/tests/gtest_rendezvous_hashing.cpp b/src/Storages/ObjectStorage/tests/gtest_rendezvous_hashing.cpp index c9fc2831b214..1df88cb6adf8 100644 --- a/src/Storages/ObjectStorage/tests/gtest_rendezvous_hashing.cpp +++ b/src/Storages/ObjectStorage/tests/gtest_rendezvous_hashing.cpp @@ -1,6 +1,7 @@ #include -#include + #include +#include using namespace DB; diff --git a/tests/integration/test_storage_iceberg_with_spark/test_deletion_vectors.py b/tests/integration/test_storage_iceberg_with_spark/test_deletion_vectors.py new file mode 100644 index 000000000000..f0a90730864e --- /dev/null +++ b/tests/integration/test_storage_iceberg_with_spark/test_deletion_vectors.py @@ -0,0 +1,644 @@ +import uuid + +import pytest + +from helpers.iceberg_utils import ( + default_upload_directory, + get_uuid_str, + get_creation_expression, +) + + +def get_array(query_result: str): + return sorted([int(x) for x in query_result.strip().split("\n") if x]) + + +def upload_table(cluster, storage_type, table_name): + default_upload_directory( + cluster, + storage_type, + f"/iceberg_data/default/{table_name}/", + f"/iceberg_data/default/{table_name}/", + ) + + +def add_equality_deletes_by_id(spark, table_name, ids): + """Commit an Iceberg equality-delete file for the given `id` values. + + Spark SQL DELETE on v3 only writes deletion vectors, so equality deletes are + produced by writing a Parquet file with Spark (correct long boxing) and + registering it via Iceberg RowDelta. + """ + jvm = spark._jvm + ice = jvm.org.apache.iceberg + table = ice.spark.Spark3Util.loadIcebergTable( + spark._jsparkSession, f"spark_catalog.default.{table_name}" + ) + + id_field_id = int(table.schema().findField("id").fieldId()) + file_name = f"eq-delete-{uuid.uuid4()}.parquet" + delete_path = table.locationProvider().newDataLocation(file_name) + + # Spark DataFrame write keeps BIGINT as Long — avoids py4j Integer boxing. + spark.createDataFrame([(int(v),) for v in ids], "id: long").coalesce(1).write.mode( + "overwrite" + ).parquet(delete_path) + + # coalesce(1) still writes a directory; pick the single part file. + hadoop_path = jvm.org.apache.hadoop.fs.Path(delete_path) + fs = hadoop_path.getFileSystem(spark.sparkContext._jsc.hadoopConfiguration()) + statuses = fs.listStatus(hadoop_path) + part_path = None + part_size = 0 + for status in statuses: + name = status.getPath().getName() + if name.startswith("part-") and name.endswith(".parquet"): + part_path = status.getPath().toString() + part_size = int(status.getLen()) + break + if part_path is None: + raise RuntimeError(f"No parquet part file written under {delete_path}") + + equality_field_ids = spark.sparkContext._gateway.new_array(jvm.int, 1) + equality_field_ids[0] = id_field_id + + delete_file = ( + ice.FileMetadata.deleteFileBuilder(table.spec()) + .ofEqualityDeletes(equality_field_ids) + .withPath(part_path) + .withFileSizeInBytes(part_size) + .withRecordCount(len(ids)) + .withFormat(ice.FileFormat.PARQUET) + .build() + ) + table.newRowDelta().addDeletes(delete_file).commit() + + +@pytest.mark.parametrize("run_on_cluster", [False, True]) +@pytest.mark.parametrize("storage_type", ["s3", "azure", "local"]) +def test_deletion_vectors(started_cluster_iceberg_with_spark, storage_type, run_on_cluster): + if storage_type == "local" and run_on_cluster: + pytest.skip("Local storage with cluster execution is not supported") + + instance = started_cluster_iceberg_with_spark.instances["node1"] + spark = started_cluster_iceberg_with_spark.spark_session + table_name = "test_deletion_vectors_" + storage_type + "_" + get_uuid_str() + deleted_ids = [2, 5, 7, 100] + + spark.sql( + f""" + CREATE TABLE {table_name} (id bigint) USING iceberg + TBLPROPERTIES ( + 'format-version' = '3', + 'write.delete.mode' = 'merge-on-read', + 'write.update.mode' = 'merge-on-read', + 'write.merge.mode' = 'merge-on-read' + ) + """ + ) + spark.sql(f"INSERT INTO {table_name} SELECT id FROM range(0, 200)") + spark.sql( + f"DELETE FROM {table_name} WHERE id IN ({', '.join(str(x) for x in deleted_ids)})" + ) + + upload_table(started_cluster_iceberg_with_spark, storage_type, table_name) + + expression = get_creation_expression( + storage_type, + table_name, + started_cluster_iceberg_with_spark, + run_on_cluster=run_on_cluster, + table_function=True, + ) + + assert int(instance.query(f"SELECT count() FROM {expression}")) == 200 - len(deleted_ids) + assert get_array(instance.query(f"SELECT id FROM {expression}")) == [ + x for x in range(200) if x not in deleted_ids + ] + + +@pytest.mark.parametrize("run_on_cluster", [False, True]) +@pytest.mark.parametrize("storage_type", ["s3", "azure", "local"]) +def test_deletion_vectors_aggregates(started_cluster_iceberg_with_spark, storage_type, run_on_cluster): + """Aggregates over Iceberg v3 tables must ignore rows covered by deletion vectors.""" + if storage_type == "local" and run_on_cluster: + pytest.skip("Local storage with cluster execution is not supported") + + instance = started_cluster_iceberg_with_spark.instances["node1"] + spark = started_cluster_iceberg_with_spark.spark_session + table_name = "test_deletion_vectors_aggregates_" + storage_type + "_" + get_uuid_str() + deleted_ids = {2, 5, 7, 50, 99} + remaining_ids = [i for i in range(100) if i not in deleted_ids] + # value = 10 * id, so sum/avg expectations stay integer-friendly where possible. + remaining_values = [10 * i for i in remaining_ids] + + spark.sql( + f""" + CREATE TABLE {table_name} (id bigint, value bigint, group_id int) USING iceberg + TBLPROPERTIES ( + 'format-version' = '3', + 'write.delete.mode' = 'merge-on-read', + 'write.update.mode' = 'merge-on-read', + 'write.merge.mode' = 'merge-on-read' + ) + """ + ) + spark.sql( + f""" + INSERT INTO {table_name} + SELECT id, 10 * id, CAST(id % 3 AS INT) + FROM range(0, 100) + """ + ) + spark.sql( + f"DELETE FROM {table_name} WHERE id IN ({', '.join(str(x) for x in sorted(deleted_ids))})" + ) + + upload_table(started_cluster_iceberg_with_spark, storage_type, table_name) + + expression = get_creation_expression( + storage_type, + table_name, + started_cluster_iceberg_with_spark, + run_on_cluster=run_on_cluster, + table_function=True, + ) + + spark_row = spark.sql( + f""" + SELECT + count(*) AS cnt, + sum(id) AS sum_id, + sum(value) AS sum_value, + min(id) AS min_id, + max(id) AS max_id, + avg(value) AS avg_value + FROM {table_name} + """ + ).collect()[0] + + expected_count = len(remaining_ids) + expected_sum_id = sum(remaining_ids) + expected_sum_value = sum(remaining_values) + expected_min_id = min(remaining_ids) + expected_max_id = max(remaining_ids) + expected_avg_value = expected_sum_value / expected_count + + assert spark_row["cnt"] == expected_count + assert spark_row["sum_id"] == expected_sum_id + assert spark_row["sum_value"] == expected_sum_value + assert spark_row["min_id"] == expected_min_id + assert spark_row["max_id"] == expected_max_id + assert abs(float(spark_row["avg_value"]) - expected_avg_value) < 1e-9 + + ch_row = instance.query( + f""" + SELECT + count(), + count(id), + sum(id), + sum(value), + min(id), + max(id), + avg(value), + uniqExact(id), + countIf(id % 2 = 0), + sumIf(value, id % 2 = 0) + FROM {expression} + """ + ).strip().split("\t") + + assert int(ch_row[0]) == expected_count + assert int(ch_row[1]) == expected_count + assert int(ch_row[2]) == expected_sum_id + assert int(ch_row[3]) == expected_sum_value + assert int(ch_row[4]) == expected_min_id + assert int(ch_row[5]) == expected_max_id + assert abs(float(ch_row[6]) - expected_avg_value) < 1e-9 + assert int(ch_row[7]) == expected_count + + expected_even_ids = [i for i in remaining_ids if i % 2 == 0] + expected_count_if = len(expected_even_ids) + expected_sum_if = sum(10 * i for i in expected_even_ids) + assert int(ch_row[8]) == expected_count_if + assert int(ch_row[9]) == expected_sum_if + + # Trivial COUNT must match the full scan once deletion vectors are applied. + assert ( + int( + instance.query( + f"SELECT count() FROM {expression}", + settings={"optimize_trivial_count_query": 1}, + ) + ) + == expected_count + ) + assert ( + int( + instance.query( + f"SELECT count() FROM {expression}", + settings={"optimize_trivial_count_query": 0}, + ) + ) + == expected_count + ) + + # GROUP BY aggregates must also exclude deleted rows. + spark_groups = { + int(row["group_id"]): (int(row["cnt"]), int(row["sum_value"])) + for row in spark.sql( + f""" + SELECT group_id, count(*) AS cnt, sum(value) AS sum_value + FROM {table_name} + GROUP BY group_id + ORDER BY group_id + """ + ).collect() + } + ch_groups = {} + for line in instance.query( + f""" + SELECT group_id, count(), sum(value) + FROM {expression} + GROUP BY group_id + ORDER BY group_id + """ + ).strip().split("\n"): + group_id, cnt, sum_value = line.split("\t") + ch_groups[int(group_id)] = (int(cnt), int(sum_value)) + + expected_groups = {} + for group_id in (0, 1, 2): + ids = [i for i in remaining_ids if i % 3 == group_id] + expected_groups[group_id] = (len(ids), sum(10 * i for i in ids)) + + assert spark_groups == expected_groups + assert ch_groups == expected_groups + + +@pytest.mark.parametrize("storage_type", ["s3", "azure", "local"]) +def test_deletion_vectors_complex(started_cluster_iceberg_with_spark, storage_type): + instance = started_cluster_iceberg_with_spark.instances["node1"] + spark = started_cluster_iceberg_with_spark.spark_session + table_name = "test_deletion_vectors_complex_" + storage_type + "_" + get_uuid_str() + + def expected_complex_ids(): + ids = list(range(20, 90)) + list(range(100, 150)) + ids += [x for x in range(200, 250) if x not in {205, 210, 220}] + return sorted(ids) + + expected_ids = expected_complex_ids() + + spark.sql( + f""" + CREATE TABLE {table_name} (id bigint, data string) USING iceberg + PARTITIONED BY (bucket(5, id)) + TBLPROPERTIES ( + 'format-version' = '3', + 'write.delete.mode' = 'merge-on-read', + 'write.update.mode' = 'merge-on-read', + 'write.merge.mode' = 'merge-on-read' + ) + """ + ) + spark.sql( + f"INSERT INTO {table_name} SELECT id, char(id + ascii('a')) FROM range(10, 100)" + ) + upload_table(started_cluster_iceberg_with_spark, storage_type, table_name) + + expression = get_creation_expression( + storage_type, + table_name, + started_cluster_iceberg_with_spark, + table_function=True, + ) + + assert int(instance.query(f"SELECT count(id) FROM {expression}")) == 90 + assert get_array(instance.query(f"SELECT id FROM {expression}")) == list(range(10, 100)) + + spark.sql(f"DELETE FROM {table_name} WHERE id < 20") + upload_table(started_cluster_iceberg_with_spark, storage_type, table_name) + assert get_array(instance.query(f"SELECT id FROM {expression}")) == list(range(20, 100)) + + spark.sql(f"DELETE FROM {table_name} WHERE id >= 90") + upload_table(started_cluster_iceberg_with_spark, storage_type, table_name) + assert get_array(instance.query(f"SELECT id FROM {expression}")) == list(range(20, 90)) + + spark.sql( + f"INSERT INTO {table_name} SELECT id, char(id + ascii('a')) FROM range(100, 200)" + ) + upload_table(started_cluster_iceberg_with_spark, storage_type, table_name) + assert get_array(instance.query(f"SELECT id FROM {expression}")) == list(range(20, 90)) + list( + range(100, 200) + ) + + spark.sql(f"DELETE FROM {table_name} WHERE id >= 150") + upload_table(started_cluster_iceberg_with_spark, storage_type, table_name) + assert get_array(instance.query(f"SELECT id FROM {expression}")) == list(range(20, 90)) + list( + range(100, 150) + ) + + spark.sql(f"ALTER TABLE {table_name} ADD COLUMNS (label string)") + spark.sql( + f""" + INSERT INTO {table_name} + SELECT id, char(id + ascii('a')), 'new' + FROM range(200, 250) + """ + ) + upload_table(started_cluster_iceberg_with_spark, storage_type, table_name) + assert get_array(instance.query(f"SELECT id FROM {expression}")) == list(range(20, 90)) + list( + range(100, 150) + ) + list(range(200, 250)) + assert int(instance.query(f"SELECT count(id) FROM {expression} WHERE label = 'new'")) == 50 + + spark.sql(f"DELETE FROM {table_name} WHERE id IN (205, 210, 220)") + upload_table(started_cluster_iceberg_with_spark, storage_type, table_name) + assert get_array(instance.query(f"SELECT id FROM {expression}")) == expected_ids + assert int(instance.query(f"SELECT count(id) FROM {expression}")) == len(expected_ids) + + spark.sql(f"UPDATE {table_name} SET label = 'updated' WHERE id = 25") + upload_table(started_cluster_iceberg_with_spark, storage_type, table_name) + assert instance.query(f"SELECT label FROM {expression} WHERE id = 25").strip() == "updated" + assert int(instance.query(f"SELECT count(id) FROM {expression} WHERE label = 'updated'")) == 1 + + spark.sql(f"CALL system.rewrite_data_files(table => '{table_name}')") + upload_table(started_cluster_iceberg_with_spark, storage_type, table_name) + assert get_array(instance.query(f"SELECT id FROM {expression}")) == expected_ids + + assert get_array( + instance.query( + f"SELECT id FROM {expression} WHERE id % 3 = 0" + ) + ) == sorted([x for x in expected_ids if x % 3 == 0]) + + +@pytest.mark.parametrize("storage_type", ["s3"]) +def test_deletion_vectors_puffin_files_cache(started_cluster_iceberg_with_spark, storage_type): + instance = started_cluster_iceberg_with_spark.instances["node1"] + spark = started_cluster_iceberg_with_spark.spark_session + table_name = "test_deletion_vectors_cache_" + storage_type + "_" + get_uuid_str() + deleted_ids = [2, 5, 7, 100] + + spark.sql( + f""" + CREATE TABLE {table_name} (id bigint) USING iceberg + TBLPROPERTIES ( + 'format-version' = '3', + 'write.delete.mode' = 'merge-on-read', + 'write.update.mode' = 'merge-on-read', + 'write.merge.mode' = 'merge-on-read' + ) + """ + ) + spark.sql(f"INSERT INTO {table_name} SELECT id FROM range(0, 200)") + spark.sql( + f"DELETE FROM {table_name} WHERE id IN ({', '.join(str(x) for x in deleted_ids)})" + ) + + upload_table(started_cluster_iceberg_with_spark, storage_type, table_name) + + expression = get_creation_expression( + storage_type, + table_name, + started_cluster_iceberg_with_spark, + table_function=True, + ) + + instance.query("SYSTEM DROP PUFFIN FILES CACHE") + + query_id1 = f"{table_name}-{uuid.uuid4()}" + query_id2 = f"{table_name}-{uuid.uuid4()}" + query_id3 = f"{table_name}-{uuid.uuid4()}" + + assert int( + instance.query( + f"SELECT count(id) FROM {expression}", + query_id=query_id1, + settings={"use_puffin_files_cache": 1}, + ) + ) == 200 - len(deleted_ids) + + assert int( + instance.query( + f"SELECT count(id) FROM {expression}", + query_id=query_id2, + settings={"use_puffin_files_cache": 1}, + ) + ) == 200 - len(deleted_ids) + + instance.query("SYSTEM FLUSH LOGS") + + assert int( + instance.query( + f"SELECT ProfileEvents['PuffinFilesCacheMisses'] FROM system.query_log WHERE query_id = '{query_id1}' AND type = 'QueryFinish'" + ) + ) > 0 + assert int( + instance.query( + f"SELECT ProfileEvents['PuffinFilesCacheHits'] FROM system.query_log WHERE query_id = '{query_id2}' AND type = 'QueryFinish'" + ) + ) > 0 + + puffin_reads_first = int( + instance.query( + f"SELECT ProfileEvents['PuffinFilesRead'] FROM system.query_log WHERE query_id = '{query_id1}' AND type = 'QueryFinish'" + ) + ) + puffin_reads_second = int( + instance.query( + f"SELECT ProfileEvents['PuffinFilesRead'] FROM system.query_log WHERE query_id = '{query_id2}' AND type = 'QueryFinish'" + ) + ) + assert puffin_reads_first > 0 + assert puffin_reads_second == 0 + + instance.query("SYSTEM DROP PUFFIN FILES CACHE") + + assert int( + instance.query( + f"SELECT count(id) FROM {expression}", + query_id=query_id3, + settings={"use_puffin_files_cache": 1}, + ) + ) == 200 - len(deleted_ids) + + instance.query("SYSTEM FLUSH LOGS") + + assert int( + instance.query( + f"SELECT ProfileEvents['PuffinFilesCacheMisses'] FROM system.query_log WHERE query_id = '{query_id3}' AND type = 'QueryFinish'" + ) + ) > int( + instance.query( + f"SELECT ProfileEvents['PuffinFilesCacheMisses'] FROM system.query_log WHERE query_id = '{query_id2}' AND type = 'QueryFinish'" + ) + ) + + +@pytest.mark.parametrize("storage_type", ["s3", "azure", "local"]) +def test_deletion_vectors_reject_mutations(started_cluster_iceberg_with_spark, storage_type): + """DELETE/UPDATE must fail closed on tables that already contain deletion vectors. + + ClickHouse mutations write parquet position-delete files, which Iceberg readers ignore for + data files that have a matching DV — so a successful mutation would silently leave rows. + """ + instance = started_cluster_iceberg_with_spark.instances["node1"] + spark = started_cluster_iceberg_with_spark.spark_session + table_name = "test_deletion_vectors_reject_mutations_" + storage_type + "_" + get_uuid_str() + + spark.sql( + f""" + CREATE TABLE {table_name} (id bigint) USING iceberg + TBLPROPERTIES ( + 'format-version' = '3', + 'write.delete.mode' = 'merge-on-read', + 'write.update.mode' = 'merge-on-read', + 'write.merge.mode' = 'merge-on-read' + ) + """ + ) + spark.sql(f"INSERT INTO {table_name} SELECT id FROM range(0, 20)") + spark.sql(f"DELETE FROM {table_name} WHERE id IN (1, 2, 3)") + + upload_table(started_cluster_iceberg_with_spark, storage_type, table_name) + + instance.query( + get_creation_expression( + storage_type, + table_name, + started_cluster_iceberg_with_spark, + table_function=False, + ) + ) + + assert int(instance.query(f"SELECT count() FROM {table_name}")) == 17 + + delete_error = instance.query_and_get_error( + f"ALTER TABLE {table_name} DELETE WHERE id = 4", + settings={"allow_insert_into_iceberg": 1}, + ) + assert "deletion vectors" in delete_error.lower() + + update_error = instance.query_and_get_error( + f"ALTER TABLE {table_name} UPDATE id = 0 WHERE id = 4", + settings={"allow_insert_into_iceberg": 1}, + ) + assert "deletion vectors" in update_error.lower() + + # Rows must be unchanged after rejected mutations. + assert int(instance.query(f"SELECT count() FROM {table_name}")) == 17 + assert get_array(instance.query(f"SELECT id FROM {table_name}")) == [ + x for x in range(20) if x not in (1, 2, 3) + ] + + +@pytest.mark.parametrize("run_on_cluster", [False, True]) +@pytest.mark.parametrize("storage_type", ["s3", "azure", "local"]) +def test_deletion_vectors_with_equality_deletes( + started_cluster_iceberg_with_spark, storage_type, run_on_cluster +): + """DV + equality deletes on the same table must keep correct survivors. + + Guards StorageObjectStorageSource transform order: DV must run before equality + FilterTransform (see gtest_deletion_vector_before_equality_filter). + """ + if storage_type == "local" and run_on_cluster: + pytest.skip("Local storage with cluster execution is not supported") + + instance = started_cluster_iceberg_with_spark.instances["node1"] + spark = started_cluster_iceberg_with_spark.spark_session + table_name = "test_dv_with_eq_" + storage_type + "_" + get_uuid_str() + + # Small unpartitioned file so DV and equality deletes both apply to the same data file. + spark.sql( + f""" + CREATE TABLE {table_name} (id bigint) USING iceberg + TBLPROPERTIES ( + 'format-version' = '3', + 'write.delete.mode' = 'merge-on-read', + 'write.update.mode' = 'merge-on-read', + 'write.merge.mode' = 'merge-on-read' + ) + """ + ) + spark.sql(f"INSERT INTO {table_name} SELECT id FROM range(0, 20)") + # Deletion vector removes file positions for these ids (values equal positions here). + spark.sql(f"DELETE FROM {table_name} WHERE id IN (2, 7)") + # Equality deletes remove by value after DV materialization. + add_equality_deletes_by_id(spark, table_name, [1, 5]) + + upload_table(started_cluster_iceberg_with_spark, storage_type, table_name) + + expression = get_creation_expression( + storage_type, + table_name, + started_cluster_iceberg_with_spark, + run_on_cluster=run_on_cluster, + table_function=True, + ) + + deleted = {1, 2, 5, 7} + expected = [x for x in range(20) if x not in deleted] + spark_ids = sorted(int(r[0]) for r in spark.sql(f"SELECT id FROM {table_name}").collect()) + assert spark_ids == expected + + assert int(instance.query(f"SELECT count() FROM {expression}")) == len(expected) + assert get_array(instance.query(f"SELECT id FROM {expression}")) == expected + + +@pytest.mark.parametrize("storage_type", ["s3", "azure"]) +def test_deletion_vectors_cluster_bucket_split(started_cluster_iceberg_with_spark, storage_type): + """icebergCluster bucket splitting must preserve DV (and clone) metadata end-to-end.""" + instance = started_cluster_iceberg_with_spark.instances["node1"] + spark = started_cluster_iceberg_with_spark.spark_session + table_name = "test_dv_cluster_bucket_" + storage_type + "_" + get_uuid_str() + deleted_ids = [2, 5, 7, 100] + + spark.sql( + f""" + CREATE TABLE {table_name} (id bigint) USING iceberg + TBLPROPERTIES ( + 'format-version' = '3', + 'write.delete.mode' = 'merge-on-read', + 'write.update.mode' = 'merge-on-read', + 'write.merge.mode' = 'merge-on-read', + 'write.parquet.row-group-size-bytes' = '1' + ) + """ + ) + spark.sql(f"INSERT INTO {table_name} SELECT id FROM range(0, 200)") + spark.sql( + f"DELETE FROM {table_name} WHERE id IN ({', '.join(str(x) for x in deleted_ids)})" + ) + + upload_table(started_cluster_iceberg_with_spark, storage_type, table_name) + + expression = get_creation_expression( + storage_type, + table_name, + started_cluster_iceberg_with_spark, + run_on_cluster=True, + table_function=True, + ) + expected = [x for x in range(200) if x not in deleted_ids] + settings = { + "cluster_table_function_split_granularity": "bucket", + "cluster_table_function_buckets_batch_size": 1, + } + + assert ( + int(instance.query(f"SELECT count() FROM {expression}", settings=settings)) + == len(expected) + ) + assert get_array(instance.query(f"SELECT id FROM {expression}", settings=settings)) == expected + # Trivial count path must match under bucket splits (need_only_count + DV cardinality). + assert ( + int( + instance.query( + f"SELECT count() FROM {expression}", + settings={**settings, "optimize_trivial_count_query": 1}, + ) + ) + == len(expected) + ) diff --git a/tests/queries/0_stateless/01271_show_privileges.reference b/tests/queries/0_stateless/01271_show_privileges.reference index d42feba5f564..e067ced12d7b 100644 --- a/tests/queries/0_stateless/01271_show_privileges.reference +++ b/tests/queries/0_stateless/01271_show_privileges.reference @@ -130,6 +130,7 @@ SYSTEM DROP MARK CACHE ['SYSTEM CLEAR MARK CACHE','SYSTEM DROP MARK','DROP MARK SYSTEM DROP ICEBERG METADATA CACHE ['SYSTEM CLEAR ICEBERG_METADATA_CACHE','SYSTEM DROP ICEBERG_METADATA_CACHE'] GLOBAL SYSTEM DROP CACHE SYSTEM DROP AVRO SCHEMA CACHE ['SYSTEM CLEAR AVRO SCHEMA CACHE','SYSTEM DROP AVRO SCHEMA CACHE','DROP AVRO SCHEMA CACHE'] GLOBAL SYSTEM DROP CACHE SYSTEM DROP PARQUET METADATA CACHE ['SYSTEM DROP PARQUET_METADATA_CACHE'] GLOBAL SYSTEM DROP CACHE +SYSTEM DROP PUFFIN FILES CACHE ['SYSTEM DROP PUFFIN_FILES_CACHE'] GLOBAL SYSTEM DROP CACHE SYSTEM PREWARM PRIMARY INDEX CACHE ['SYSTEM PREWARM PRIMARY INDEX','PREWARM PRIMARY INDEX CACHE','PREWARM PRIMARY INDEX'] GLOBAL SYSTEM DROP CACHE SYSTEM DROP PRIMARY INDEX CACHE ['SYSTEM CLEAR PRIMARY INDEX CACHE','SYSTEM DROP PRIMARY INDEX','DROP PRIMARY INDEX CACHE','DROP PRIMARY INDEX'] GLOBAL SYSTEM DROP CACHE SYSTEM DROP UNCOMPRESSED CACHE ['SYSTEM CLEAR UNCOMPRESSED CACHE','SYSTEM DROP UNCOMPRESSED','DROP UNCOMPRESSED CACHE','DROP UNCOMPRESSED'] GLOBAL SYSTEM DROP CACHE diff --git a/tests/queries/0_stateless/04077_puffin_happy_path.reference b/tests/queries/0_stateless/04077_puffin_happy_path.reference index 66cf85ce28f3..dc6d6814d154 100644 --- a/tests/queries/0_stateless/04077_puffin_happy_path.reference +++ b/tests/queries/0_stateless/04077_puffin_happy_path.reference @@ -19,6 +19,12 @@ deletion-vector-v1 -1 -1 4 44 ['cardinality','referenced-data-file'] ['2','/dat /data/table/part-00000.parquet [2,5] /data/table/part-00000.parquet 2 /data/table/part-00000.parquet 5 +--- mixed_blob_types.puffin --- +apache-datasketches-theta-v1 -1 -1 4 16 [] [] +deletion-vector-v1 -1 -1 20 44 ['cardinality','referenced-data-file'] ['2','/data/table/part-00000.parquet'] +/data/table/part-00000.parquet [2,5] +/data/table/part-00000.parquet 2 +/data/table/part-00000.parquet 5 --- dense_range_100k.puffin --- 100000 0 99999 --- subset without deleted_rows --- diff --git a/tests/queries/0_stateless/04077_puffin_happy_path.sh b/tests/queries/0_stateless/04077_puffin_happy_path.sh index eedc30b57880..d86afa63a37c 100755 --- a/tests/queries/0_stateless/04077_puffin_happy_path.sh +++ b/tests/queries/0_stateless/04077_puffin_happy_path.sh @@ -32,6 +32,7 @@ run_happy_path() { run_happy_path "spark_deletion_vector.puffin" "$DATA/spark_deletion_vector.puffin" run_happy_path "compressed_footer.puffin" "$DATA/compressed_footer.puffin" run_happy_path "file_properties_ok.puffin" "$DATA/file_properties_ok.puffin" +run_happy_path "mixed_blob_types.puffin" "$DATA/mixed_blob_types.puffin" echo "--- dense_range_100k.puffin ---" $CLICKHOUSE_LOCAL -q " diff --git a/tests/queries/0_stateless/04117_parser_system_query_variants.reference b/tests/queries/0_stateless/04117_parser_system_query_variants.reference index 2ffe7e9fb91f..3b81cbe7ae02 100644 --- a/tests/queries/0_stateless/04117_parser_system_query_variants.reference +++ b/tests/queries/0_stateless/04117_parser_system_query_variants.reference @@ -15,6 +15,8 @@ SYSTEM CLEAR QUERY CONDITION CACHE SYSTEM CLEAR COMPILED EXPRESSION CACHE SYSTEM CLEAR ICEBERG METADATA CACHE SYSTEM CLEAR PARQUET METADATA CACHE +SYSTEM CLEAR PUFFIN FILES CACHE +SYSTEM CLEAR PUFFIN FILES CACHE SYSTEM CLEAR FILESYSTEM CACHE SYSTEM CLEAR DISTRIBUTED CACHE SYSTEM CLEAR DISTRIBUTED CACHE CONNECTIONS ON CLUSTER cluster diff --git a/tests/queries/0_stateless/04117_parser_system_query_variants.sql b/tests/queries/0_stateless/04117_parser_system_query_variants.sql index c48bb44537f9..bb68fdee5aa5 100644 --- a/tests/queries/0_stateless/04117_parser_system_query_variants.sql +++ b/tests/queries/0_stateless/04117_parser_system_query_variants.sql @@ -25,6 +25,8 @@ EXPLAIN SYNTAX SYSTEM DROP QUERY CONDITION CACHE; EXPLAIN SYNTAX SYSTEM DROP COMPILED EXPRESSION CACHE; EXPLAIN SYNTAX SYSTEM DROP ICEBERG METADATA CACHE; EXPLAIN SYNTAX SYSTEM DROP PARQUET METADATA CACHE; +EXPLAIN SYNTAX SYSTEM DROP PUFFIN FILES CACHE; +EXPLAIN SYNTAX SYSTEM DROP PUFFIN_FILES_CACHE; EXPLAIN SYNTAX SYSTEM DROP FILESYSTEM CACHE; EXPLAIN SYNTAX SYSTEM DROP DISTRIBUTED CACHE; EXPLAIN SYNTAX SYSTEM DROP DISTRIBUTED CACHE CONNECTIONS ON CLUSTER cluster; diff --git a/tests/queries/0_stateless/04261_iceberg_deletion_vector.reference b/tests/queries/0_stateless/04261_iceberg_deletion_vector.reference new file mode 100644 index 000000000000..fe7ad97c7e35 --- /dev/null +++ b/tests/queries/0_stateless/04261_iceberg_deletion_vector.reference @@ -0,0 +1,197 @@ +196 +0 +1 +3 +4 +6 +8 +9 +10 +11 +12 +13 +14 +15 +16 +17 +18 +19 +20 +21 +22 +23 +24 +25 +26 +27 +28 +29 +30 +31 +32 +33 +34 +35 +36 +37 +38 +39 +40 +41 +42 +43 +44 +45 +46 +47 +48 +49 +50 +51 +52 +53 +54 +55 +56 +57 +58 +59 +60 +61 +62 +63 +64 +65 +66 +67 +68 +69 +70 +71 +72 +73 +74 +75 +76 +77 +78 +79 +80 +81 +82 +83 +84 +85 +86 +87 +88 +89 +90 +91 +92 +93 +94 +95 +96 +97 +98 +99 +101 +102 +103 +104 +105 +106 +107 +108 +109 +110 +111 +112 +113 +114 +115 +116 +117 +118 +119 +120 +121 +122 +123 +124 +125 +126 +127 +128 +129 +130 +131 +132 +133 +134 +135 +136 +137 +138 +139 +140 +141 +142 +143 +144 +145 +146 +147 +148 +149 +150 +151 +152 +153 +154 +155 +156 +157 +158 +159 +160 +161 +162 +163 +164 +165 +166 +167 +168 +169 +170 +171 +172 +173 +174 +175 +176 +177 +178 +179 +180 +181 +182 +183 +184 +185 +186 +187 +188 +189 +190 +191 +192 +193 +194 +195 +196 +197 +198 +199 diff --git a/tests/queries/0_stateless/04261_iceberg_deletion_vector.sh b/tests/queries/0_stateless/04261_iceberg_deletion_vector.sh new file mode 100755 index 000000000000..b09a64d3e88d --- /dev/null +++ b/tests/queries/0_stateless/04261_iceberg_deletion_vector.sh @@ -0,0 +1,16 @@ +#!/usr/bin/env bash +# Tags: no-fasttest + +CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source=../shell_config.sh +. "$CURDIR"/../shell_config.sh + +TABLE_PATH="${CURDIR}/data_minio/dv_puffin_warehouse/default/dv_puffin_source" + +$CLICKHOUSE_LOCAL -q " +SELECT count() FROM icebergLocal('${TABLE_PATH}') +" + +$CLICKHOUSE_LOCAL -q " +SELECT id FROM icebergLocal('${TABLE_PATH}') ORDER BY id +" diff --git a/tests/queries/0_stateless/04262_iceberg_deletion_vector_complex.reference b/tests/queries/0_stateless/04262_iceberg_deletion_vector_complex.reference new file mode 100644 index 000000000000..32bd071ad3c5 --- /dev/null +++ b/tests/queries/0_stateless/04262_iceberg_deletion_vector_complex.reference @@ -0,0 +1,172 @@ +167 +47 +1 +updated +119 +20 +21 +22 +23 +24 +25 +26 +27 +28 +29 +30 +31 +32 +33 +34 +35 +36 +37 +38 +39 +40 +41 +42 +43 +44 +45 +46 +47 +48 +49 +50 +51 +52 +53 +54 +55 +56 +57 +58 +59 +60 +61 +62 +63 +64 +65 +66 +67 +68 +69 +70 +71 +72 +73 +74 +75 +76 +77 +78 +79 +80 +81 +82 +83 +84 +85 +86 +87 +88 +89 +100 +101 +102 +103 +104 +105 +106 +107 +108 +109 +110 +111 +112 +113 +114 +115 +116 +117 +118 +119 +120 +121 +122 +123 +124 +125 +126 +127 +128 +129 +130 +131 +132 +133 +134 +135 +136 +137 +138 +139 +140 +141 +142 +143 +144 +145 +146 +147 +148 +149 +200 +201 +202 +203 +204 +206 +207 +208 +209 +211 +212 +213 +214 +215 +216 +217 +218 +219 +221 +222 +223 +224 +225 +226 +227 +228 +229 +230 +231 +232 +233 +234 +235 +236 +237 +238 +239 +240 +241 +242 +243 +244 +245 +246 +247 +248 +249 diff --git a/tests/queries/0_stateless/04262_iceberg_deletion_vector_complex.sh b/tests/queries/0_stateless/04262_iceberg_deletion_vector_complex.sh new file mode 100755 index 000000000000..0fe7da881e31 --- /dev/null +++ b/tests/queries/0_stateless/04262_iceberg_deletion_vector_complex.sh @@ -0,0 +1,32 @@ +#!/usr/bin/env bash +# Tags: no-fasttest + +CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source=../shell_config.sh +. "$CURDIR"/../shell_config.sh + +TABLE_PATH="${CURDIR}/data_minio/dv_puffin_warehouse/default/dv_puffin_complex" + +$CLICKHOUSE_LOCAL -q " +SELECT count(id) FROM icebergLocal('${TABLE_PATH}') +" + +$CLICKHOUSE_LOCAL -q " +SELECT count(id) FROM icebergLocal('${TABLE_PATH}') WHERE label = 'new' +" + +$CLICKHOUSE_LOCAL -q " +SELECT count(id) FROM icebergLocal('${TABLE_PATH}') WHERE label = 'updated' +" + +$CLICKHOUSE_LOCAL -q " +SELECT label FROM icebergLocal('${TABLE_PATH}') WHERE id = 25 +" + +$CLICKHOUSE_LOCAL -q " +SELECT count(id) FROM icebergLocal('${TABLE_PATH}') WHERE label IS NULL +" + +$CLICKHOUSE_LOCAL -q " +SELECT id FROM icebergLocal('${TABLE_PATH}') ORDER BY id +" diff --git a/tests/queries/0_stateless/04263_iceberg_puffin_files_cache.reference b/tests/queries/0_stateless/04263_iceberg_puffin_files_cache.reference new file mode 100644 index 000000000000..50a04fbf6628 --- /dev/null +++ b/tests/queries/0_stateless/04263_iceberg_puffin_files_cache.reference @@ -0,0 +1,19 @@ +196 +PuffinFilesCacheMisses 1 +PuffinFilesRead 2 +196 +PuffinFilesCacheHits 1 +PuffinFilesCacheMisses 1 +PuffinFilesRead 2 +196 +PuffinFilesCacheHits 1 +PuffinFilesCacheMisses 2 +PuffinFilesRead 4 +196 +PuffinFilesCacheHits 1 +PuffinFilesCacheMisses 2 +PuffinFilesRead 6 +196 +PuffinFilesCacheHits 1 +PuffinFilesCacheMisses 2 +PuffinFilesRead 8 diff --git a/tests/queries/0_stateless/04263_iceberg_puffin_files_cache.sh b/tests/queries/0_stateless/04263_iceberg_puffin_files_cache.sh new file mode 100755 index 000000000000..f9565b8cf146 --- /dev/null +++ b/tests/queries/0_stateless/04263_iceberg_puffin_files_cache.sh @@ -0,0 +1,65 @@ +#!/usr/bin/env bash +# Tags: no-fasttest, no-parallel, no-parallel-replicas, no-random-settings +# no-fasttest: depends on local iceberg fixture +# no-parallel: cache is system-wide and tests can affect each other in unexpected way +# no-parallel-replicas: profile events are not available on the second replica +# no-random-settings: we need to test the interaction of specific setting combinations + +CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source=../shell_config.sh +. "$CURDIR"/../shell_config.sh + +TABLE_PATH="${CURDIR}/data_minio/dv_puffin_warehouse/default/dv_puffin_source" + +$CLICKHOUSE_LOCAL -q " +SYSTEM DROP PUFFIN FILES CACHE; + +SELECT count(id) +FROM icebergLocal('${TABLE_PATH}') +SETTINGS use_puffin_files_cache = 1; + +SELECT event, value +FROM system.events +WHERE event IN ('PuffinFilesCacheHits', 'PuffinFilesCacheMisses', 'PuffinFilesRead') +ORDER BY event; + +SELECT count(id) +FROM icebergLocal('${TABLE_PATH}') +SETTINGS use_puffin_files_cache = 1; + +SELECT event, value +FROM system.events +WHERE event IN ('PuffinFilesCacheHits', 'PuffinFilesCacheMisses', 'PuffinFilesRead') +ORDER BY event; + +SYSTEM DROP PUFFIN FILES CACHE; + +SELECT count(id) +FROM icebergLocal('${TABLE_PATH}') +SETTINGS use_puffin_files_cache = 1; + +SELECT event, value +FROM system.events +WHERE event IN ('PuffinFilesCacheHits', 'PuffinFilesCacheMisses', 'PuffinFilesRead') +ORDER BY event; + +SYSTEM DROP PUFFIN FILES CACHE; + +SELECT count(id) +FROM icebergLocal('${TABLE_PATH}') +SETTINGS use_puffin_files_cache = 0; + +SELECT event, value +FROM system.events +WHERE event IN ('PuffinFilesCacheHits', 'PuffinFilesCacheMisses', 'PuffinFilesRead') +ORDER BY event; + +SELECT count(id) +FROM icebergLocal('${TABLE_PATH}') +SETTINGS use_puffin_files_cache = 0; + +SELECT event, value +FROM system.events +WHERE event IN ('PuffinFilesCacheHits', 'PuffinFilesCacheMisses', 'PuffinFilesRead') +ORDER BY event; +" diff --git a/tests/queries/0_stateless/04337_iceberg_v3_row_lineage_reserved_field_id.sh b/tests/queries/0_stateless/04337_iceberg_v3_row_lineage_reserved_field_id.sh index c56f43b2a637..13f56710a84f 100755 --- a/tests/queries/0_stateless/04337_iceberg_v3_row_lineage_reserved_field_id.sh +++ b/tests/queries/0_stateless/04337_iceberg_v3_row_lineage_reserved_field_id.sh @@ -56,13 +56,17 @@ ${CLICKHOUSE_CLIENT} --query "SELECT x FROM icebergLocal('${ICEBERG_TABLE_PATH}' # Conversely, 2147483447 (Integer.MAX_VALUE - 200) is the highest field id a table may use, i.e. # NOT reserved. An unmapped column with that id is a genuine schema mismatch and must still be # rejected, so the reserved-range check must be strictly greater-than. +# +# Insert at least two distinct values so allow_experimental_iceberg_read_optimization cannot treat +# `x` as a constant from Iceberg column stats and take the need-only-count path (which never runs +# SchemaConverter / field-id mapping checks). ICEBERG_TABLE_PATH_UNMAPPED="${CLICKHOUSE_USER_FILES}/lakehouses/${CLICKHOUSE_DATABASE}_v3_unmapped" rm -rf "${ICEBERG_TABLE_PATH_UNMAPPED}" ${CLICKHOUSE_CLIENT} --query " SET allow_experimental_insert_into_iceberg = 1; CREATE TABLE t_v3_unmapped (x Int32) ENGINE = IcebergLocal('${ICEBERG_TABLE_PATH_UNMAPPED}'); - INSERT INTO t_v3_unmapped (x) VALUES (1); + INSERT INTO t_v3_unmapped (x) VALUES (1), (2); " DATAFILE_UNMAPPED=$(ls "${ICEBERG_TABLE_PATH_UNMAPPED}"/data/*.parquet 2>/dev/null | head -1) @@ -77,7 +81,7 @@ x = pa.field("x", pa.int32(), nullable=False, metadata={b"PARQUET:field_id": b"1 # 2147483447 = Integer.MAX_VALUE - 200: the highest id a table may use, so NOT reserved. extra = pa.field("extra", pa.int64(), nullable=True, metadata={b"PARQUET:field_id": b"2147483447"}) table = pa.table( - {"x": pa.array([1], pa.int32()), "extra": pa.array([0], pa.int64())}, + {"x": pa.array([1, 2], pa.int32()), "extra": pa.array([0, 1], pa.int64())}, schema=pa.schema([x, extra]), ) pq.write_table(table, path) diff --git a/tests/queries/0_stateless/04549_puffin_allow_seeks_off.sh b/tests/queries/0_stateless/04549_puffin_allow_seeks_off.sh index 238205a43c66..8ae4bfff924e 100755 --- a/tests/queries/0_stateless/04549_puffin_allow_seeks_off.sh +++ b/tests/queries/0_stateless/04549_puffin_allow_seeks_off.sh @@ -23,6 +23,8 @@ SETTINGS input_format_allow_seeks = 0 " # Non-seekable path must reject a wrong leading magic before buffering the rest of the stream. +# Oversized valid-magic streams are covered by gtest_puffin_non_seekable_buffer_limit (absolute +# buffer ceiling is ~2 GiB + footer; too large for a CI fixture). echo "--- non-Puffin magic allow_seeks=0 ---" NOT_PUFFIN="${CLICKHOUSE_TMP}/04549_not_puffin.bin" # Prefix is wrong; trailing bytes would only matter if we buffered first then validated. diff --git a/tests/queries/0_stateless/04613_puffin_metadata_type_errors.reference b/tests/queries/0_stateless/04613_puffin_metadata_type_errors.reference index 1bb9169aca6f..9ee286b03479 100644 --- a/tests/queries/0_stateless/04613_puffin_metadata_type_errors.reference +++ b/tests/queries/0_stateless/04613_puffin_metadata_type_errors.reference @@ -28,6 +28,10 @@ Cannot parse Puffin footer JSON Cannot parse Puffin footer JSON --- dv_with_compression_codec.puffin --- must omit 'compression-codec' +--- dv_snapshot_id_not_minus_one.puffin --- +snapshot-id and sequence-number must be -1 +--- dv_sequence_number_not_minus_one.puffin --- +snapshot-id and sequence-number must be -1 --- invalid_cardinality_non_numeric.puffin --- property 'cardinality' must be an unsigned integer --- invalid_cardinality_negative.puffin --- diff --git a/tests/queries/0_stateless/04613_puffin_metadata_type_errors.sh b/tests/queries/0_stateless/04613_puffin_metadata_type_errors.sh index a4915b9de77e..fb6e5429f7fe 100755 --- a/tests/queries/0_stateless/04613_puffin_metadata_type_errors.sh +++ b/tests/queries/0_stateless/04613_puffin_metadata_type_errors.sh @@ -47,6 +47,12 @@ done launch "$id" meta "$DATA/dv_with_compression_codec.puffin" "must omit 'compression-codec'" id=$((id + 1)) +for f in dv_snapshot_id_not_minus_one dv_sequence_number_not_minus_one +do + launch "$id" meta "$DATA/$f.puffin" 'snapshot-id and sequence-number must be -1' + id=$((id + 1)) +done + for f in invalid_cardinality_non_numeric invalid_cardinality_negative do launch "$id" meta "$DATA/$f.puffin" "property 'cardinality' must be an unsigned integer" diff --git a/tests/queries/0_stateless/04656_count_from_files_cache_row_policy.reference b/tests/queries/0_stateless/04656_count_from_files_cache_row_policy.reference new file mode 100644 index 000000000000..256b5a9ab73d --- /dev/null +++ b/tests/queries/0_stateless/04656_count_from_files_cache_row_policy.reference @@ -0,0 +1,2 @@ +with_policy 5 +after_drop_policy 10 diff --git a/tests/queries/0_stateless/04656_count_from_files_cache_row_policy.sh b/tests/queries/0_stateless/04656_count_from_files_cache_row_policy.sh new file mode 100755 index 000000000000..dce2e16028e2 --- /dev/null +++ b/tests/queries/0_stateless/04656_count_from_files_cache_row_policy.sh @@ -0,0 +1,30 @@ +#!/usr/bin/env bash + +CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source=../shell_config.sh +. "$CURDIR"/../shell_config.sh + +# Filtered reads (row policy) must not populate count-from-files cache with a reduced +# cardinality that later unrestricted counts would reuse. + +$CLICKHOUSE_CLIENT -q " +DROP TABLE IF EXISTS ${CLICKHOUSE_DATABASE}.t_04656; +DROP ROW POLICY IF EXISTS p_04656 ON ${CLICKHOUSE_DATABASE}.t_04656; + +CREATE TABLE ${CLICKHOUSE_DATABASE}.t_04656 (x UInt8) ENGINE = File(TSV); +INSERT INTO ${CLICKHOUSE_DATABASE}.t_04656 SELECT number FROM numbers(10); + +CREATE ROW POLICY p_04656 ON ${CLICKHOUSE_DATABASE}.t_04656 USING x < 5 TO ALL; + +SELECT 'with_policy', count() +FROM ${CLICKHOUSE_DATABASE}.t_04656 +SETTINGS use_cache_for_count_from_files = 1, optimize_count_from_files = 1; + +DROP ROW POLICY p_04656 ON ${CLICKHOUSE_DATABASE}.t_04656; + +SELECT 'after_drop_policy', count() +FROM ${CLICKHOUSE_DATABASE}.t_04656 +SETTINGS use_cache_for_count_from_files = 1, optimize_count_from_files = 1; + +DROP TABLE ${CLICKHOUSE_DATABASE}.t_04656; +" diff --git a/tests/queries/0_stateless/04671_iceberg_v3_mutations_rejected.reference b/tests/queries/0_stateless/04671_iceberg_v3_mutations_rejected.reference new file mode 100644 index 000000000000..7dc9f22985a6 --- /dev/null +++ b/tests/queries/0_stateless/04671_iceberg_v3_mutations_rejected.reference @@ -0,0 +1,3 @@ +SUPPORT_IS_DISABLED +SUPPORT_IS_DISABLED +3 [1,2,3] diff --git a/tests/queries/0_stateless/04671_iceberg_v3_mutations_rejected.sh b/tests/queries/0_stateless/04671_iceberg_v3_mutations_rejected.sh new file mode 100755 index 000000000000..e5e9b0f9b045 --- /dev/null +++ b/tests/queries/0_stateless/04671_iceberg_v3_mutations_rejected.sh @@ -0,0 +1,41 @@ +#!/usr/bin/env bash +# Tags: no-fasttest + +# Iceberg v3 writers must not add new position-delete files. ClickHouse mutations +# still write parquet position deletes, so DELETE/UPDATE on a format-version-3 +# table (even without existing deletion vectors) must fail closed before any +# object writes. + +# Force quieter logs: CI often pre-sets CLICKHOUSE_CLIENT_SERVER_LOGS_LEVEL=warning, +# and ${VAR:-error} would keep that. IcebergMetadata logs a Warning when reading +# CH-written v3 metadata (v1 `schema` missing → v2 `schemas` fallback). +CLICKHOUSE_CLIENT_SERVER_LOGS_LEVEL=error + +CUR_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source=../shell_config.sh +. "$CUR_DIR"/../shell_config.sh + +TABLE="t_${CLICKHOUSE_DATABASE}_${RANDOM}" +TABLE_PATH="${USER_FILES_PATH}/${TABLE}/" + +trap 'rm -rf "${TABLE_PATH}" 2>/dev/null' EXIT + +${CLICKHOUSE_CLIENT} --query "DROP TABLE IF EXISTS ${TABLE}" +${CLICKHOUSE_CLIENT} --query " + CREATE TABLE ${TABLE} (c0 Int32) + ENGINE = IcebergLocal('${TABLE_PATH}', 'Parquet') + SETTINGS iceberg_format_version = 3 +" + +${CLICKHOUSE_CLIENT} --allow_insert_into_iceberg=1 --query "INSERT INTO ${TABLE} VALUES (1), (2), (3)" + +# Fail closed: no position-delete files may be written for v3. +${CLICKHOUSE_CLIENT} --allow_insert_into_iceberg=1 --query \ + "ALTER TABLE ${TABLE} DELETE WHERE c0 = 1" 2>&1 | grep -o 'SUPPORT_IS_DISABLED' | head -n1 +${CLICKHOUSE_CLIENT} --allow_insert_into_iceberg=1 --query \ + "ALTER TABLE ${TABLE} UPDATE c0 = 10 WHERE c0 = 2" 2>&1 | grep -o 'SUPPORT_IS_DISABLED' | head -n1 + +# Rows must be unchanged (rejection happened before object writes). +${CLICKHOUSE_CLIENT} --query "SELECT count(), groupArray(c0) FROM (SELECT c0 FROM ${TABLE} ORDER BY c0)" + +${CLICKHOUSE_CLIENT} --query "DROP TABLE IF EXISTS ${TABLE}" diff --git a/tests/queries/0_stateless/04672_iceberg_dv_count_metadata_only.reference b/tests/queries/0_stateless/04672_iceberg_dv_count_metadata_only.reference new file mode 100644 index 000000000000..b3e9d2ae831b --- /dev/null +++ b/tests/queries/0_stateless/04672_iceberg_dv_count_metadata_only.reference @@ -0,0 +1,4 @@ +196 +count: metadata_only +optimize_off: decoded +row_group_spans: metadata_only diff --git a/tests/queries/0_stateless/04672_iceberg_dv_count_metadata_only.sh b/tests/queries/0_stateless/04672_iceberg_dv_count_metadata_only.sh new file mode 100755 index 000000000000..3b08a4e870dd --- /dev/null +++ b/tests/queries/0_stateless/04672_iceberg_dv_count_metadata_only.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +# Tags: no-fasttest + +# Regression: SELECT count() over Iceberg deletion vectors must use Parquet +# needOnlyCount (footer / row-group metadata) plus roaring range cardinality, +# not decode data pages. Previously hasAttachedDeletes cleared need_only_count +# for any DV, making DeletionVectorTransform's const-count path unreachable. +# Bucketed / per-row-group spans use the same const-chunk + row_num_offset shape. + +CUR_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source=../shell_config.sh +. "$CUR_DIR"/../shell_config.sh + +TABLE_PATH="${CUR_DIR}/data_minio/dv_puffin_warehouse/default/dv_puffin_source" + +profile_sum() +{ + local event="$1" + awk -v e="$event" '$0 ~ e ":" { sum += $(NF-1) } END { print sum+0 }' +} + +assert_metadata_only() +{ + local label="$1" + local output="$2" + local decode read_rg + decode=$(printf '%s\n' "$output" | profile_sum ParquetDecodingTasks) + read_rg=$(printf '%s\n' "$output" | profile_sum ParquetReadRowGroups) + if [ "$decode" -eq 0 ] && [ "$read_rg" -eq 0 ]; then + echo "${label}: metadata_only" + else + echo "${label}: decoded (ParquetDecodingTasks=${decode} ParquetReadRowGroups=${read_rg})" + fi +} + +# Correct surviving row count (4 deletes from 200). +$CLICKHOUSE_LOCAL -q " +SELECT count() FROM icebergLocal('${TABLE_PATH}') +SETTINGS + optimize_count_from_files = 1, + use_cache_for_count_from_files = 0 +" + +# Count must not decode column pages / open row-group readers. +out=$($CLICKHOUSE_LOCAL --print-profile-events -q " +SELECT count() FROM icebergLocal('${TABLE_PATH}') +SETTINGS + optimize_count_from_files = 1, + use_cache_for_count_from_files = 0 +" 2>&1) +assert_metadata_only "count" "$out" + +# Contrast: disabling count-from-files must decode (proves counters are live). +out_full=$($CLICKHOUSE_LOCAL --print-profile-events -q " +SELECT count() FROM icebergLocal('${TABLE_PATH}') +SETTINGS + optimize_count_from_files = 0, + use_cache_for_count_from_files = 0 +" 2>&1) +decode_full=$(printf '%s\n' "$out_full" | profile_sum ParquetDecodingTasks) +if [ "$decode_full" -gt 0 ]; then + echo "optimize_off: decoded" +else + echo "optimize_off: unexpectedly_metadata_only" +fi + +# Per-row-group needOnlyCount spans (same chunk shape as cluster bucket splits): +# absolute row_num_offset + const defaults, adjusted by DV range cardinality. +out_spans=$($CLICKHOUSE_LOCAL --print-profile-events -q " +SELECT count() FROM icebergLocal('${TABLE_PATH}') +SETTINGS + optimize_count_from_files = 1, + use_cache_for_count_from_files = 0, + max_block_size = 1 +" 2>&1) +assert_metadata_only "row_group_spans" "$out_spans" diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/README.md b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/README.md new file mode 100644 index 000000000000..f4cf3a4c173b --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/README.md @@ -0,0 +1,35 @@ +# Iceberg v3 deletion vector fixtures + +Generated with `generate_iceberg_dv_fixture.py` (Spark + Iceberg 1.9.0). + +Warehouse: `dv_puffin_warehouse/` + +## `default/dv_puffin_source` + +Simple table with column `id BIGINT`: + +- 200 rows (`id` from 0 to 199) +- Deleted rows via Puffin deletion vector: 2, 5, 7, 100 + +## `default/dv_puffin_complex` + +Table with columns `id BIGINT`, `data STRING`, `label STRING` and multiple snapshots: + +- Initial insert `id` 10-99, then deletes `id < 20` and `id >= 90` +- Insert `id` 100-199, delete `id >= 150` +- Schema evolution (`label` column), insert `id` 200-249 with `label = 'new'` +- Delete `id` in (205, 210, 220), update `id = 25` to `label = 'updated'` +- Spark `rewrite_data_files` compaction + +Regenerate: + +```bash +python3 generate_iceberg_dv_fixture.py +``` + +Query paths for tests: + +```text +data_minio/dv_puffin_warehouse/default/dv_puffin_source +data_minio/dv_puffin_warehouse/default/dv_puffin_complex +``` diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/00000-14-ba106366-1379-49df-a585-047546b341d9-00001-deletes.puffin b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/00000-14-ba106366-1379-49df-a585-047546b341d9-00001-deletes.puffin new file mode 100644 index 000000000000..956042e717a5 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/00000-14-ba106366-1379-49df-a585-047546b341d9-00001-deletes.puffin differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/00000-18-0da8af70-7e2e-4ced-935b-d2b8ef79eb76-00001-deletes.puffin b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/00000-18-0da8af70-7e2e-4ced-935b-d2b8ef79eb76-00001-deletes.puffin new file mode 100644 index 000000000000..94aa1644cfd8 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/00000-18-0da8af70-7e2e-4ced-935b-d2b8ef79eb76-00001-deletes.puffin differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/00000-20-04b90311-de4e-4d07-9633-95670a571224-00001-deletes.puffin b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/00000-20-04b90311-de4e-4d07-9633-95670a571224-00001-deletes.puffin new file mode 100644 index 000000000000..ce4a05ab2431 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/00000-20-04b90311-de4e-4d07-9633-95670a571224-00001-deletes.puffin differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/00000-6-74340066-9c57-43b9-9112-c569f91929b9-00001-deletes.puffin b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/00000-6-74340066-9c57-43b9-9112-c569f91929b9-00001-deletes.puffin new file mode 100644 index 000000000000..7bb98d451f28 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/00000-6-74340066-9c57-43b9-9112-c569f91929b9-00001-deletes.puffin differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/00000-9-dcc19f77-3570-494e-b336-f4d3301291b4-00001-deletes.puffin b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/00000-9-dcc19f77-3570-494e-b336-f4d3301291b4-00001-deletes.puffin new file mode 100644 index 000000000000..7df1cec9ff2a Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/00000-9-dcc19f77-3570-494e-b336-f4d3301291b4-00001-deletes.puffin differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=0/00000-11-499dd379-3429-4dc8-9457-9b0d5a6a6e9b-0-00005.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=0/00000-11-499dd379-3429-4dc8-9457-9b0d5a6a6e9b-0-00005.parquet new file mode 100644 index 000000000000..fb364a125c2f Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=0/00000-11-499dd379-3429-4dc8-9457-9b0d5a6a6e9b-0-00005.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=0/00000-16-03e76970-c0de-4110-9928-e74531d8b125-0-00005.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=0/00000-16-03e76970-c0de-4110-9928-e74531d8b125-0-00005.parquet new file mode 100644 index 000000000000..d9b655282bf8 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=0/00000-16-03e76970-c0de-4110-9928-e74531d8b125-0-00005.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=0/00000-20-04b90311-de4e-4d07-9633-95670a571224-00001.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=0/00000-20-04b90311-de4e-4d07-9633-95670a571224-00001.parquet new file mode 100644 index 000000000000..6dfde23d8eae Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=0/00000-20-04b90311-de4e-4d07-9633-95670a571224-00001.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=0/00000-25-ee6fe6d5-2f4f-47e6-ad97-00c53284ff87-0-00001.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=0/00000-25-ee6fe6d5-2f4f-47e6-ad97-00c53284ff87-0-00001.parquet new file mode 100644 index 000000000000..97c37bb1ab5d Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=0/00000-25-ee6fe6d5-2f4f-47e6-ad97-00c53284ff87-0-00001.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=0/00000-4-fe8ff9da-4256-45aa-9ed1-48200445069f-0-00005.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=0/00000-4-fe8ff9da-4256-45aa-9ed1-48200445069f-0-00005.parquet new file mode 100644 index 000000000000..f5da4d7105d6 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=0/00000-4-fe8ff9da-4256-45aa-9ed1-48200445069f-0-00005.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=1/00000-11-499dd379-3429-4dc8-9457-9b0d5a6a6e9b-0-00001.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=1/00000-11-499dd379-3429-4dc8-9457-9b0d5a6a6e9b-0-00001.parquet new file mode 100644 index 000000000000..43f7e9c338c6 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=1/00000-11-499dd379-3429-4dc8-9457-9b0d5a6a6e9b-0-00001.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=1/00000-16-03e76970-c0de-4110-9928-e74531d8b125-0-00001.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=1/00000-16-03e76970-c0de-4110-9928-e74531d8b125-0-00001.parquet new file mode 100644 index 000000000000..ada984ceed8d Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=1/00000-16-03e76970-c0de-4110-9928-e74531d8b125-0-00001.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=1/00000-24-0b9f04c6-aebb-4cfc-a3b7-3f2a1034a6dc-0-00001.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=1/00000-24-0b9f04c6-aebb-4cfc-a3b7-3f2a1034a6dc-0-00001.parquet new file mode 100644 index 000000000000..0f39927f6c7b Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=1/00000-24-0b9f04c6-aebb-4cfc-a3b7-3f2a1034a6dc-0-00001.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=1/00000-4-fe8ff9da-4256-45aa-9ed1-48200445069f-0-00001.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=1/00000-4-fe8ff9da-4256-45aa-9ed1-48200445069f-0-00001.parquet new file mode 100644 index 000000000000..76b701bf1e2b Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=1/00000-4-fe8ff9da-4256-45aa-9ed1-48200445069f-0-00001.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=2/00000-11-499dd379-3429-4dc8-9457-9b0d5a6a6e9b-0-00004.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=2/00000-11-499dd379-3429-4dc8-9457-9b0d5a6a6e9b-0-00004.parquet new file mode 100644 index 000000000000..8b4c38ecebe0 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=2/00000-11-499dd379-3429-4dc8-9457-9b0d5a6a6e9b-0-00004.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=2/00000-16-03e76970-c0de-4110-9928-e74531d8b125-0-00004.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=2/00000-16-03e76970-c0de-4110-9928-e74531d8b125-0-00004.parquet new file mode 100644 index 000000000000..cb037bcb4efa Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=2/00000-16-03e76970-c0de-4110-9928-e74531d8b125-0-00004.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=2/00000-23-292ac3e8-88a4-482a-9a88-e08ad8bdd2e6-0-00001.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=2/00000-23-292ac3e8-88a4-482a-9a88-e08ad8bdd2e6-0-00001.parquet new file mode 100644 index 000000000000..690316c0bced Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=2/00000-23-292ac3e8-88a4-482a-9a88-e08ad8bdd2e6-0-00001.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=2/00000-4-fe8ff9da-4256-45aa-9ed1-48200445069f-0-00004.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=2/00000-4-fe8ff9da-4256-45aa-9ed1-48200445069f-0-00004.parquet new file mode 100644 index 000000000000..56882346e595 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=2/00000-4-fe8ff9da-4256-45aa-9ed1-48200445069f-0-00004.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=3/00000-11-499dd379-3429-4dc8-9457-9b0d5a6a6e9b-0-00002.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=3/00000-11-499dd379-3429-4dc8-9457-9b0d5a6a6e9b-0-00002.parquet new file mode 100644 index 000000000000..f9661fe1b8f3 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=3/00000-11-499dd379-3429-4dc8-9457-9b0d5a6a6e9b-0-00002.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=3/00000-16-03e76970-c0de-4110-9928-e74531d8b125-0-00002.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=3/00000-16-03e76970-c0de-4110-9928-e74531d8b125-0-00002.parquet new file mode 100644 index 000000000000..c9baf49eb9c9 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=3/00000-16-03e76970-c0de-4110-9928-e74531d8b125-0-00002.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=3/00000-21-55307a4a-f583-48b6-983a-c2993f261fec-0-00001.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=3/00000-21-55307a4a-f583-48b6-983a-c2993f261fec-0-00001.parquet new file mode 100644 index 000000000000..4a5ea28ead0c Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=3/00000-21-55307a4a-f583-48b6-983a-c2993f261fec-0-00001.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=3/00000-4-fe8ff9da-4256-45aa-9ed1-48200445069f-0-00002.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=3/00000-4-fe8ff9da-4256-45aa-9ed1-48200445069f-0-00002.parquet new file mode 100644 index 000000000000..82ba5ea76d37 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=3/00000-4-fe8ff9da-4256-45aa-9ed1-48200445069f-0-00002.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=4/00000-11-499dd379-3429-4dc8-9457-9b0d5a6a6e9b-0-00003.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=4/00000-11-499dd379-3429-4dc8-9457-9b0d5a6a6e9b-0-00003.parquet new file mode 100644 index 000000000000..852ad728cd69 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=4/00000-11-499dd379-3429-4dc8-9457-9b0d5a6a6e9b-0-00003.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=4/00000-16-03e76970-c0de-4110-9928-e74531d8b125-0-00003.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=4/00000-16-03e76970-c0de-4110-9928-e74531d8b125-0-00003.parquet new file mode 100644 index 000000000000..4d8d2c59eee2 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=4/00000-16-03e76970-c0de-4110-9928-e74531d8b125-0-00003.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=4/00000-22-cbc4c1d2-88dd-4d8a-a31a-6db198e852d4-0-00001.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=4/00000-22-cbc4c1d2-88dd-4d8a-a31a-6db198e852d4-0-00001.parquet new file mode 100644 index 000000000000..9fcb8cee529f Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=4/00000-22-cbc4c1d2-88dd-4d8a-a31a-6db198e852d4-0-00001.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=4/00000-4-fe8ff9da-4256-45aa-9ed1-48200445069f-0-00003.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=4/00000-4-fe8ff9da-4256-45aa-9ed1-48200445069f-0-00003.parquet new file mode 100644 index 000000000000..50b6c3ed85e6 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/data/id_bucket=4/00000-4-fe8ff9da-4256-45aa-9ed1-48200445069f-0-00003.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/0c03abfe-66a8-43ab-a765-a9e8072b59ed-m0.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/0c03abfe-66a8-43ab-a765-a9e8072b59ed-m0.avro new file mode 100644 index 000000000000..50388b8a2ebb Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/0c03abfe-66a8-43ab-a765-a9e8072b59ed-m0.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/23ca5b2b-d3f2-44b4-8494-f0aadd4e0b34-m0.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/23ca5b2b-d3f2-44b4-8494-f0aadd4e0b34-m0.avro new file mode 100644 index 000000000000..2289c0aaeea3 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/23ca5b2b-d3f2-44b4-8494-f0aadd4e0b34-m0.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/321000ac-b08e-430f-9c79-e40d5e4aa42d-m0.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/321000ac-b08e-430f-9c79-e40d5e4aa42d-m0.avro new file mode 100644 index 000000000000..0cce769e50f0 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/321000ac-b08e-430f-9c79-e40d5e4aa42d-m0.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/321000ac-b08e-430f-9c79-e40d5e4aa42d-m1.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/321000ac-b08e-430f-9c79-e40d5e4aa42d-m1.avro new file mode 100644 index 000000000000..c9facfde4c04 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/321000ac-b08e-430f-9c79-e40d5e4aa42d-m1.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/39ec1ceb-2dbe-4ab7-ab4e-0d18bed2d2da-m0.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/39ec1ceb-2dbe-4ab7-ab4e-0d18bed2d2da-m0.avro new file mode 100644 index 000000000000..2f84bfe01b74 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/39ec1ceb-2dbe-4ab7-ab4e-0d18bed2d2da-m0.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/41caec64-2cad-46f5-a966-dd75e3635abc-m0.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/41caec64-2cad-46f5-a966-dd75e3635abc-m0.avro new file mode 100644 index 000000000000..3f7df28d60e9 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/41caec64-2cad-46f5-a966-dd75e3635abc-m0.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/41caec64-2cad-46f5-a966-dd75e3635abc-m1.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/41caec64-2cad-46f5-a966-dd75e3635abc-m1.avro new file mode 100644 index 000000000000..59d5e1f574b4 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/41caec64-2cad-46f5-a966-dd75e3635abc-m1.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/5b66c606-c296-4fe4-a5f8-d8be592f6b96-m0.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/5b66c606-c296-4fe4-a5f8-d8be592f6b96-m0.avro new file mode 100644 index 000000000000..9084391f0e62 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/5b66c606-c296-4fe4-a5f8-d8be592f6b96-m0.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/5b66c606-c296-4fe4-a5f8-d8be592f6b96-m1.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/5b66c606-c296-4fe4-a5f8-d8be592f6b96-m1.avro new file mode 100644 index 000000000000..c42c203e2b30 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/5b66c606-c296-4fe4-a5f8-d8be592f6b96-m1.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/5b66c606-c296-4fe4-a5f8-d8be592f6b96-m2.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/5b66c606-c296-4fe4-a5f8-d8be592f6b96-m2.avro new file mode 100644 index 000000000000..97a79ed3f6d9 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/5b66c606-c296-4fe4-a5f8-d8be592f6b96-m2.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/5b66c606-c296-4fe4-a5f8-d8be592f6b96-m3.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/5b66c606-c296-4fe4-a5f8-d8be592f6b96-m3.avro new file mode 100644 index 000000000000..e96733a33279 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/5b66c606-c296-4fe4-a5f8-d8be592f6b96-m3.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/5b66c606-c296-4fe4-a5f8-d8be592f6b96-m4.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/5b66c606-c296-4fe4-a5f8-d8be592f6b96-m4.avro new file mode 100644 index 000000000000..1825a92f8fa4 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/5b66c606-c296-4fe4-a5f8-d8be592f6b96-m4.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/6a19b33f-2be5-4073-820c-82cb1c7d76cb-m0.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/6a19b33f-2be5-4073-820c-82cb1c7d76cb-m0.avro new file mode 100644 index 000000000000..b41643648c61 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/6a19b33f-2be5-4073-820c-82cb1c7d76cb-m0.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/b68e784a-6b9c-4894-827d-c56978599ecd-m0.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/b68e784a-6b9c-4894-827d-c56978599ecd-m0.avro new file mode 100644 index 000000000000..2dbdc71cc3cd Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/b68e784a-6b9c-4894-827d-c56978599ecd-m0.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/c21c4572-6006-4649-85b1-47e82e73a396-m0.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/c21c4572-6006-4649-85b1-47e82e73a396-m0.avro new file mode 100644 index 000000000000..4f57b9fff1fe Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/c21c4572-6006-4649-85b1-47e82e73a396-m0.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-212327459647764229-1-c21c4572-6006-4649-85b1-47e82e73a396.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-212327459647764229-1-c21c4572-6006-4649-85b1-47e82e73a396.avro new file mode 100644 index 000000000000..1c6208295989 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-212327459647764229-1-c21c4572-6006-4649-85b1-47e82e73a396.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2216958009875447676-1-b68e784a-6b9c-4894-827d-c56978599ecd.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2216958009875447676-1-b68e784a-6b9c-4894-827d-c56978599ecd.avro new file mode 100644 index 000000000000..619989a8a328 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2216958009875447676-1-b68e784a-6b9c-4894-827d-c56978599ecd.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2245381059480343696-1-23ca5b2b-d3f2-44b4-8494-f0aadd4e0b34.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2245381059480343696-1-23ca5b2b-d3f2-44b4-8494-f0aadd4e0b34.avro new file mode 100644 index 000000000000..62699c0aa26d Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2245381059480343696-1-23ca5b2b-d3f2-44b4-8494-f0aadd4e0b34.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-243132775900853210-1-0c03abfe-66a8-43ab-a765-a9e8072b59ed.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-243132775900853210-1-0c03abfe-66a8-43ab-a765-a9e8072b59ed.avro new file mode 100644 index 000000000000..151ec4194d68 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-243132775900853210-1-0c03abfe-66a8-43ab-a765-a9e8072b59ed.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-399037466005519109-1-6a19b33f-2be5-4073-820c-82cb1c7d76cb.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-399037466005519109-1-6a19b33f-2be5-4073-820c-82cb1c7d76cb.avro new file mode 100644 index 000000000000..9ca00ca319c2 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-399037466005519109-1-6a19b33f-2be5-4073-820c-82cb1c7d76cb.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-7131460395349610904-1-321000ac-b08e-430f-9c79-e40d5e4aa42d.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-7131460395349610904-1-321000ac-b08e-430f-9c79-e40d5e4aa42d.avro new file mode 100644 index 000000000000..929e6598ea9f Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-7131460395349610904-1-321000ac-b08e-430f-9c79-e40d5e4aa42d.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8352556540874311077-1-39ec1ceb-2dbe-4ab7-ab4e-0d18bed2d2da.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8352556540874311077-1-39ec1ceb-2dbe-4ab7-ab4e-0d18bed2d2da.avro new file mode 100644 index 000000000000..070f738c8a08 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8352556540874311077-1-39ec1ceb-2dbe-4ab7-ab4e-0d18bed2d2da.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8541702041949031574-1-5b66c606-c296-4fe4-a5f8-d8be592f6b96.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8541702041949031574-1-5b66c606-c296-4fe4-a5f8-d8be592f6b96.avro new file mode 100644 index 000000000000..8911f605816f Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8541702041949031574-1-5b66c606-c296-4fe4-a5f8-d8be592f6b96.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-9221420601809522049-1-41caec64-2cad-46f5-a966-dd75e3635abc.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-9221420601809522049-1-41caec64-2cad-46f5-a966-dd75e3635abc.avro new file mode 100644 index 000000000000..09ad8a80111b Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-9221420601809522049-1-41caec64-2cad-46f5-a966-dd75e3635abc.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v1.metadata.json b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v1.metadata.json new file mode 100644 index 000000000000..ee75c0491a42 --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v1.metadata.json @@ -0,0 +1 @@ +{"format-version":3,"table-uuid":"00ae7d2e-f37d-4599-83fc-756434b2c1fe","location":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex","last-sequence-number":0,"last-updated-ms":1783931355607,"last-column-id":2,"current-schema-id":0,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[{"name":"id_bucket","transform":"bucket[5]","source-id":1,"field-id":1000}]}],"last-partition-id":1000,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"owner":"iantonspb","write.merge.mode":"merge-on-read","write.update.mode":"merge-on-read","write.delete.mode":"merge-on-read","write.parquet.compression-codec":"zstd"},"current-snapshot-id":null,"next-row-id":0,"refs":{},"snapshots":[],"statistics":[],"partition-statistics":[],"snapshot-log":[],"metadata-log":[]} \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v10.metadata.json b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v10.metadata.json new file mode 100644 index 000000000000..e6b202777546 --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v10.metadata.json @@ -0,0 +1 @@ +{"format-version":3,"table-uuid":"00ae7d2e-f37d-4599-83fc-756434b2c1fe","location":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex","last-sequence-number":8,"last-updated-ms":1783931358480,"last-column-id":3,"current-schema-id":1,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"}]},{"type":"struct","schema-id":1,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"},{"id":3,"name":"label","required":false,"type":"string"}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[{"name":"id_bucket","transform":"bucket[5]","source-id":1,"field-id":1000}]}],"last-partition-id":1000,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"owner":"iantonspb","write.merge.mode":"merge-on-read","write.update.mode":"merge-on-read","write.delete.mode":"merge-on-read","write.parquet.compression-codec":"zstd"},"current-snapshot-id":9221420601809522049,"next-row-id":241,"refs":{"main":{"snapshot-id":9221420601809522049,"type":"branch"}},"snapshots":[{"sequence-number":1,"snapshot-id":8352556540874311077,"timestamp-ms":1783931356056,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"90","added-files-size":"3887","changed-partition-count":"5","total-records":"90","total-files-size":"3887","total-data-files":"5","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8352556540874311077-1-39ec1ceb-2dbe-4ab7-ab4e-0d18bed2d2da.avro","schema-id":0,"first-row-id":0,"added-rows":90},{"sequence-number":2,"snapshot-id":2245381059480343696,"parent-snapshot-id":8352556540874311077,"timestamp-ms":1783931356399,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","added-dvs":"3","added-files-size":"120","added-position-deletes":"10","changed-partition-count":"3","total-records":"90","total-files-size":"4007","total-data-files":"5","total-delete-files":"3","total-position-deletes":"10","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2245381059480343696-1-23ca5b2b-d3f2-44b4-8494-f0aadd4e0b34.avro","schema-id":0,"first-row-id":90,"added-rows":0},{"sequence-number":3,"snapshot-id":7131460395349610904,"parent-snapshot-id":2245381059480343696,"timestamp-ms":1783931356769,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","removed-delete-files":"2","added-dvs":"3","removed-dvs":"2","added-files-size":"133","removed-files-size":"81","added-position-deletes":"15","removed-position-deletes":"5","changed-partition-count":"3","total-records":"90","total-files-size":"4059","total-data-files":"5","total-delete-files":"4","total-position-deletes":"20","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-7131460395349610904-1-321000ac-b08e-430f-9c79-e40d5e4aa42d.avro","schema-id":0,"first-row-id":90,"added-rows":0},{"sequence-number":4,"snapshot-id":243132775900853210,"parent-snapshot-id":7131460395349610904,"timestamp-ms":1783931357067,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"100","added-files-size":"3936","changed-partition-count":"5","total-records":"190","total-files-size":"7995","total-data-files":"10","total-delete-files":"4","total-position-deletes":"20","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-243132775900853210-1-0c03abfe-66a8-43ab-a765-a9e8072b59ed.avro","schema-id":0,"first-row-id":90,"added-rows":100},{"sequence-number":5,"snapshot-id":212327459647764229,"parent-snapshot-id":243132775900853210,"timestamp-ms":1783931357425,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"5","added-dvs":"5","added-files-size":"195","added-position-deletes":"50","changed-partition-count":"5","total-records":"190","total-files-size":"8190","total-data-files":"10","total-delete-files":"9","total-position-deletes":"70","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-212327459647764229-1-c21c4572-6006-4649-85b1-47e82e73a396.avro","schema-id":0,"first-row-id":190,"added-rows":0},{"sequence-number":6,"snapshot-id":2216958009875447676,"parent-snapshot-id":212327459647764229,"timestamp-ms":1783931357776,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"50","added-files-size":"5103","changed-partition-count":"5","total-records":"240","total-files-size":"13293","total-data-files":"15","total-delete-files":"9","total-position-deletes":"70","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2216958009875447676-1-b68e784a-6b9c-4894-827d-c56978599ecd.avro","schema-id":1,"first-row-id":190,"added-rows":50},{"sequence-number":7,"snapshot-id":399037466005519109,"parent-snapshot-id":2216958009875447676,"timestamp-ms":1783931358118,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","added-dvs":"3","added-files-size":"126","added-position-deletes":"3","changed-partition-count":"3","total-records":"240","total-files-size":"13419","total-data-files":"15","total-delete-files":"12","total-position-deletes":"73","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-399037466005519109-1-6a19b33f-2be5-4073-820c-82cb1c7d76cb.avro","schema-id":1,"first-row-id":240,"added-rows":0},{"sequence-number":8,"snapshot-id":9221420601809522049,"parent-snapshot-id":399037466005519109,"timestamp-ms":1783931358480,"summary":{"operation":"overwrite","spark.app.id":"local-1783931350618","added-data-files":"1","added-delete-files":"1","added-dvs":"1","added-records":"1","added-files-size":"1011","added-position-deletes":"1","changed-partition-count":"1","total-records":"241","total-files-size":"14430","total-data-files":"16","total-delete-files":"13","total-position-deletes":"74","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-9221420601809522049-1-41caec64-2cad-46f5-a966-dd75e3635abc.avro","schema-id":1,"first-row-id":240,"added-rows":1}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1783931356056,"snapshot-id":8352556540874311077},{"timestamp-ms":1783931356399,"snapshot-id":2245381059480343696},{"timestamp-ms":1783931356769,"snapshot-id":7131460395349610904},{"timestamp-ms":1783931357067,"snapshot-id":243132775900853210},{"timestamp-ms":1783931357425,"snapshot-id":212327459647764229},{"timestamp-ms":1783931357776,"snapshot-id":2216958009875447676},{"timestamp-ms":1783931358118,"snapshot-id":399037466005519109},{"timestamp-ms":1783931358480,"snapshot-id":9221420601809522049}],"metadata-log":[{"timestamp-ms":1783931355607,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v1.metadata.json"},{"timestamp-ms":1783931356056,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v2.metadata.json"},{"timestamp-ms":1783931356399,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v3.metadata.json"},{"timestamp-ms":1783931356769,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v4.metadata.json"},{"timestamp-ms":1783931357067,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v5.metadata.json"},{"timestamp-ms":1783931357425,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v6.metadata.json"},{"timestamp-ms":1783931357478,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v7.metadata.json"},{"timestamp-ms":1783931357776,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v8.metadata.json"},{"timestamp-ms":1783931358118,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v9.metadata.json"}]} \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v11.metadata.json b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v11.metadata.json new file mode 100644 index 000000000000..a29d02234e63 --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v11.metadata.json @@ -0,0 +1 @@ +{"format-version":3,"table-uuid":"00ae7d2e-f37d-4599-83fc-756434b2c1fe","location":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex","last-sequence-number":9,"last-updated-ms":1783931359048,"last-column-id":3,"current-schema-id":1,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"}]},{"type":"struct","schema-id":1,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"},{"id":3,"name":"label","required":false,"type":"string"}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[{"name":"id_bucket","transform":"bucket[5]","source-id":1,"field-id":1000}]}],"last-partition-id":1000,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"owner":"iantonspb","write.merge.mode":"merge-on-read","write.update.mode":"merge-on-read","write.delete.mode":"merge-on-read","write.parquet.compression-codec":"zstd"},"current-snapshot-id":8541702041949031574,"next-row-id":408,"refs":{"main":{"snapshot-id":8541702041949031574,"type":"branch"}},"snapshots":[{"sequence-number":1,"snapshot-id":8352556540874311077,"timestamp-ms":1783931356056,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"90","added-files-size":"3887","changed-partition-count":"5","total-records":"90","total-files-size":"3887","total-data-files":"5","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8352556540874311077-1-39ec1ceb-2dbe-4ab7-ab4e-0d18bed2d2da.avro","schema-id":0,"first-row-id":0,"added-rows":90},{"sequence-number":2,"snapshot-id":2245381059480343696,"parent-snapshot-id":8352556540874311077,"timestamp-ms":1783931356399,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","added-dvs":"3","added-files-size":"120","added-position-deletes":"10","changed-partition-count":"3","total-records":"90","total-files-size":"4007","total-data-files":"5","total-delete-files":"3","total-position-deletes":"10","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2245381059480343696-1-23ca5b2b-d3f2-44b4-8494-f0aadd4e0b34.avro","schema-id":0,"first-row-id":90,"added-rows":0},{"sequence-number":3,"snapshot-id":7131460395349610904,"parent-snapshot-id":2245381059480343696,"timestamp-ms":1783931356769,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","removed-delete-files":"2","added-dvs":"3","removed-dvs":"2","added-files-size":"133","removed-files-size":"81","added-position-deletes":"15","removed-position-deletes":"5","changed-partition-count":"3","total-records":"90","total-files-size":"4059","total-data-files":"5","total-delete-files":"4","total-position-deletes":"20","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-7131460395349610904-1-321000ac-b08e-430f-9c79-e40d5e4aa42d.avro","schema-id":0,"first-row-id":90,"added-rows":0},{"sequence-number":4,"snapshot-id":243132775900853210,"parent-snapshot-id":7131460395349610904,"timestamp-ms":1783931357067,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"100","added-files-size":"3936","changed-partition-count":"5","total-records":"190","total-files-size":"7995","total-data-files":"10","total-delete-files":"4","total-position-deletes":"20","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-243132775900853210-1-0c03abfe-66a8-43ab-a765-a9e8072b59ed.avro","schema-id":0,"first-row-id":90,"added-rows":100},{"sequence-number":5,"snapshot-id":212327459647764229,"parent-snapshot-id":243132775900853210,"timestamp-ms":1783931357425,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"5","added-dvs":"5","added-files-size":"195","added-position-deletes":"50","changed-partition-count":"5","total-records":"190","total-files-size":"8190","total-data-files":"10","total-delete-files":"9","total-position-deletes":"70","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-212327459647764229-1-c21c4572-6006-4649-85b1-47e82e73a396.avro","schema-id":0,"first-row-id":190,"added-rows":0},{"sequence-number":6,"snapshot-id":2216958009875447676,"parent-snapshot-id":212327459647764229,"timestamp-ms":1783931357776,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"50","added-files-size":"5103","changed-partition-count":"5","total-records":"240","total-files-size":"13293","total-data-files":"15","total-delete-files":"9","total-position-deletes":"70","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2216958009875447676-1-b68e784a-6b9c-4894-827d-c56978599ecd.avro","schema-id":1,"first-row-id":190,"added-rows":50},{"sequence-number":7,"snapshot-id":399037466005519109,"parent-snapshot-id":2216958009875447676,"timestamp-ms":1783931358118,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","added-dvs":"3","added-files-size":"126","added-position-deletes":"3","changed-partition-count":"3","total-records":"240","total-files-size":"13419","total-data-files":"15","total-delete-files":"12","total-position-deletes":"73","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-399037466005519109-1-6a19b33f-2be5-4073-820c-82cb1c7d76cb.avro","schema-id":1,"first-row-id":240,"added-rows":0},{"sequence-number":8,"snapshot-id":9221420601809522049,"parent-snapshot-id":399037466005519109,"timestamp-ms":1783931358480,"summary":{"operation":"overwrite","spark.app.id":"local-1783931350618","added-data-files":"1","added-delete-files":"1","added-dvs":"1","added-records":"1","added-files-size":"1011","added-position-deletes":"1","changed-partition-count":"1","total-records":"241","total-files-size":"14430","total-data-files":"16","total-delete-files":"13","total-position-deletes":"74","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-9221420601809522049-1-41caec64-2cad-46f5-a966-dd75e3635abc.avro","schema-id":1,"first-row-id":240,"added-rows":1},{"sequence-number":9,"snapshot-id":8541702041949031574,"parent-snapshot-id":9221420601809522049,"timestamp-ms":1783931359048,"summary":{"operation":"replace","added-data-files":"5","deleted-data-files":"16","added-records":"167","deleted-records":"241","added-files-size":"5886","removed-files-size":"13895","changed-partition-count":"5","total-records":"167","total-files-size":"6421","total-data-files":"5","total-delete-files":"13","total-position-deletes":"74","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8541702041949031574-1-5b66c606-c296-4fe4-a5f8-d8be592f6b96.avro","schema-id":1,"first-row-id":241,"added-rows":167}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1783931356056,"snapshot-id":8352556540874311077},{"timestamp-ms":1783931356399,"snapshot-id":2245381059480343696},{"timestamp-ms":1783931356769,"snapshot-id":7131460395349610904},{"timestamp-ms":1783931357067,"snapshot-id":243132775900853210},{"timestamp-ms":1783931357425,"snapshot-id":212327459647764229},{"timestamp-ms":1783931357776,"snapshot-id":2216958009875447676},{"timestamp-ms":1783931358118,"snapshot-id":399037466005519109},{"timestamp-ms":1783931358480,"snapshot-id":9221420601809522049},{"timestamp-ms":1783931359048,"snapshot-id":8541702041949031574}],"metadata-log":[{"timestamp-ms":1783931355607,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v1.metadata.json"},{"timestamp-ms":1783931356056,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v2.metadata.json"},{"timestamp-ms":1783931356399,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v3.metadata.json"},{"timestamp-ms":1783931356769,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v4.metadata.json"},{"timestamp-ms":1783931357067,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v5.metadata.json"},{"timestamp-ms":1783931357425,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v6.metadata.json"},{"timestamp-ms":1783931357478,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v7.metadata.json"},{"timestamp-ms":1783931357776,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v8.metadata.json"},{"timestamp-ms":1783931358118,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v9.metadata.json"},{"timestamp-ms":1783931358480,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v10.metadata.json"}]} \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v2.metadata.json b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v2.metadata.json new file mode 100644 index 000000000000..be4f8957698b --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v2.metadata.json @@ -0,0 +1 @@ +{"format-version":3,"table-uuid":"00ae7d2e-f37d-4599-83fc-756434b2c1fe","location":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex","last-sequence-number":1,"last-updated-ms":1783931356056,"last-column-id":2,"current-schema-id":0,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[{"name":"id_bucket","transform":"bucket[5]","source-id":1,"field-id":1000}]}],"last-partition-id":1000,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"owner":"iantonspb","write.merge.mode":"merge-on-read","write.update.mode":"merge-on-read","write.delete.mode":"merge-on-read","write.parquet.compression-codec":"zstd"},"current-snapshot-id":8352556540874311077,"next-row-id":90,"refs":{"main":{"snapshot-id":8352556540874311077,"type":"branch"}},"snapshots":[{"sequence-number":1,"snapshot-id":8352556540874311077,"timestamp-ms":1783931356056,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"90","added-files-size":"3887","changed-partition-count":"5","total-records":"90","total-files-size":"3887","total-data-files":"5","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8352556540874311077-1-39ec1ceb-2dbe-4ab7-ab4e-0d18bed2d2da.avro","schema-id":0,"first-row-id":0,"added-rows":90}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1783931356056,"snapshot-id":8352556540874311077}],"metadata-log":[{"timestamp-ms":1783931355607,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v1.metadata.json"}]} \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v3.metadata.json b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v3.metadata.json new file mode 100644 index 000000000000..ec655a2cfe44 --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v3.metadata.json @@ -0,0 +1 @@ +{"format-version":3,"table-uuid":"00ae7d2e-f37d-4599-83fc-756434b2c1fe","location":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex","last-sequence-number":2,"last-updated-ms":1783931356399,"last-column-id":2,"current-schema-id":0,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[{"name":"id_bucket","transform":"bucket[5]","source-id":1,"field-id":1000}]}],"last-partition-id":1000,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"owner":"iantonspb","write.merge.mode":"merge-on-read","write.update.mode":"merge-on-read","write.delete.mode":"merge-on-read","write.parquet.compression-codec":"zstd"},"current-snapshot-id":2245381059480343696,"next-row-id":90,"refs":{"main":{"snapshot-id":2245381059480343696,"type":"branch"}},"snapshots":[{"sequence-number":1,"snapshot-id":8352556540874311077,"timestamp-ms":1783931356056,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"90","added-files-size":"3887","changed-partition-count":"5","total-records":"90","total-files-size":"3887","total-data-files":"5","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8352556540874311077-1-39ec1ceb-2dbe-4ab7-ab4e-0d18bed2d2da.avro","schema-id":0,"first-row-id":0,"added-rows":90},{"sequence-number":2,"snapshot-id":2245381059480343696,"parent-snapshot-id":8352556540874311077,"timestamp-ms":1783931356399,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","added-dvs":"3","added-files-size":"120","added-position-deletes":"10","changed-partition-count":"3","total-records":"90","total-files-size":"4007","total-data-files":"5","total-delete-files":"3","total-position-deletes":"10","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2245381059480343696-1-23ca5b2b-d3f2-44b4-8494-f0aadd4e0b34.avro","schema-id":0,"first-row-id":90,"added-rows":0}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1783931356056,"snapshot-id":8352556540874311077},{"timestamp-ms":1783931356399,"snapshot-id":2245381059480343696}],"metadata-log":[{"timestamp-ms":1783931355607,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v1.metadata.json"},{"timestamp-ms":1783931356056,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v2.metadata.json"}]} \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v4.metadata.json b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v4.metadata.json new file mode 100644 index 000000000000..7de46c7bff57 --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v4.metadata.json @@ -0,0 +1 @@ +{"format-version":3,"table-uuid":"00ae7d2e-f37d-4599-83fc-756434b2c1fe","location":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex","last-sequence-number":3,"last-updated-ms":1783931356769,"last-column-id":2,"current-schema-id":0,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[{"name":"id_bucket","transform":"bucket[5]","source-id":1,"field-id":1000}]}],"last-partition-id":1000,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"owner":"iantonspb","write.merge.mode":"merge-on-read","write.update.mode":"merge-on-read","write.delete.mode":"merge-on-read","write.parquet.compression-codec":"zstd"},"current-snapshot-id":7131460395349610904,"next-row-id":90,"refs":{"main":{"snapshot-id":7131460395349610904,"type":"branch"}},"snapshots":[{"sequence-number":1,"snapshot-id":8352556540874311077,"timestamp-ms":1783931356056,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"90","added-files-size":"3887","changed-partition-count":"5","total-records":"90","total-files-size":"3887","total-data-files":"5","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8352556540874311077-1-39ec1ceb-2dbe-4ab7-ab4e-0d18bed2d2da.avro","schema-id":0,"first-row-id":0,"added-rows":90},{"sequence-number":2,"snapshot-id":2245381059480343696,"parent-snapshot-id":8352556540874311077,"timestamp-ms":1783931356399,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","added-dvs":"3","added-files-size":"120","added-position-deletes":"10","changed-partition-count":"3","total-records":"90","total-files-size":"4007","total-data-files":"5","total-delete-files":"3","total-position-deletes":"10","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2245381059480343696-1-23ca5b2b-d3f2-44b4-8494-f0aadd4e0b34.avro","schema-id":0,"first-row-id":90,"added-rows":0},{"sequence-number":3,"snapshot-id":7131460395349610904,"parent-snapshot-id":2245381059480343696,"timestamp-ms":1783931356769,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","removed-delete-files":"2","added-dvs":"3","removed-dvs":"2","added-files-size":"133","removed-files-size":"81","added-position-deletes":"15","removed-position-deletes":"5","changed-partition-count":"3","total-records":"90","total-files-size":"4059","total-data-files":"5","total-delete-files":"4","total-position-deletes":"20","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-7131460395349610904-1-321000ac-b08e-430f-9c79-e40d5e4aa42d.avro","schema-id":0,"first-row-id":90,"added-rows":0}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1783931356056,"snapshot-id":8352556540874311077},{"timestamp-ms":1783931356399,"snapshot-id":2245381059480343696},{"timestamp-ms":1783931356769,"snapshot-id":7131460395349610904}],"metadata-log":[{"timestamp-ms":1783931355607,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v1.metadata.json"},{"timestamp-ms":1783931356056,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v2.metadata.json"},{"timestamp-ms":1783931356399,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v3.metadata.json"}]} \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v5.metadata.json b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v5.metadata.json new file mode 100644 index 000000000000..c118c7839b1d --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v5.metadata.json @@ -0,0 +1 @@ +{"format-version":3,"table-uuid":"00ae7d2e-f37d-4599-83fc-756434b2c1fe","location":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex","last-sequence-number":4,"last-updated-ms":1783931357067,"last-column-id":2,"current-schema-id":0,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[{"name":"id_bucket","transform":"bucket[5]","source-id":1,"field-id":1000}]}],"last-partition-id":1000,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"owner":"iantonspb","write.merge.mode":"merge-on-read","write.update.mode":"merge-on-read","write.delete.mode":"merge-on-read","write.parquet.compression-codec":"zstd"},"current-snapshot-id":243132775900853210,"next-row-id":190,"refs":{"main":{"snapshot-id":243132775900853210,"type":"branch"}},"snapshots":[{"sequence-number":1,"snapshot-id":8352556540874311077,"timestamp-ms":1783931356056,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"90","added-files-size":"3887","changed-partition-count":"5","total-records":"90","total-files-size":"3887","total-data-files":"5","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8352556540874311077-1-39ec1ceb-2dbe-4ab7-ab4e-0d18bed2d2da.avro","schema-id":0,"first-row-id":0,"added-rows":90},{"sequence-number":2,"snapshot-id":2245381059480343696,"parent-snapshot-id":8352556540874311077,"timestamp-ms":1783931356399,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","added-dvs":"3","added-files-size":"120","added-position-deletes":"10","changed-partition-count":"3","total-records":"90","total-files-size":"4007","total-data-files":"5","total-delete-files":"3","total-position-deletes":"10","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2245381059480343696-1-23ca5b2b-d3f2-44b4-8494-f0aadd4e0b34.avro","schema-id":0,"first-row-id":90,"added-rows":0},{"sequence-number":3,"snapshot-id":7131460395349610904,"parent-snapshot-id":2245381059480343696,"timestamp-ms":1783931356769,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","removed-delete-files":"2","added-dvs":"3","removed-dvs":"2","added-files-size":"133","removed-files-size":"81","added-position-deletes":"15","removed-position-deletes":"5","changed-partition-count":"3","total-records":"90","total-files-size":"4059","total-data-files":"5","total-delete-files":"4","total-position-deletes":"20","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-7131460395349610904-1-321000ac-b08e-430f-9c79-e40d5e4aa42d.avro","schema-id":0,"first-row-id":90,"added-rows":0},{"sequence-number":4,"snapshot-id":243132775900853210,"parent-snapshot-id":7131460395349610904,"timestamp-ms":1783931357067,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"100","added-files-size":"3936","changed-partition-count":"5","total-records":"190","total-files-size":"7995","total-data-files":"10","total-delete-files":"4","total-position-deletes":"20","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-243132775900853210-1-0c03abfe-66a8-43ab-a765-a9e8072b59ed.avro","schema-id":0,"first-row-id":90,"added-rows":100}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1783931356056,"snapshot-id":8352556540874311077},{"timestamp-ms":1783931356399,"snapshot-id":2245381059480343696},{"timestamp-ms":1783931356769,"snapshot-id":7131460395349610904},{"timestamp-ms":1783931357067,"snapshot-id":243132775900853210}],"metadata-log":[{"timestamp-ms":1783931355607,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v1.metadata.json"},{"timestamp-ms":1783931356056,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v2.metadata.json"},{"timestamp-ms":1783931356399,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v3.metadata.json"},{"timestamp-ms":1783931356769,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v4.metadata.json"}]} \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v6.metadata.json b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v6.metadata.json new file mode 100644 index 000000000000..0efae360a449 --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v6.metadata.json @@ -0,0 +1 @@ +{"format-version":3,"table-uuid":"00ae7d2e-f37d-4599-83fc-756434b2c1fe","location":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex","last-sequence-number":5,"last-updated-ms":1783931357425,"last-column-id":2,"current-schema-id":0,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[{"name":"id_bucket","transform":"bucket[5]","source-id":1,"field-id":1000}]}],"last-partition-id":1000,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"owner":"iantonspb","write.merge.mode":"merge-on-read","write.update.mode":"merge-on-read","write.delete.mode":"merge-on-read","write.parquet.compression-codec":"zstd"},"current-snapshot-id":212327459647764229,"next-row-id":190,"refs":{"main":{"snapshot-id":212327459647764229,"type":"branch"}},"snapshots":[{"sequence-number":1,"snapshot-id":8352556540874311077,"timestamp-ms":1783931356056,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"90","added-files-size":"3887","changed-partition-count":"5","total-records":"90","total-files-size":"3887","total-data-files":"5","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8352556540874311077-1-39ec1ceb-2dbe-4ab7-ab4e-0d18bed2d2da.avro","schema-id":0,"first-row-id":0,"added-rows":90},{"sequence-number":2,"snapshot-id":2245381059480343696,"parent-snapshot-id":8352556540874311077,"timestamp-ms":1783931356399,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","added-dvs":"3","added-files-size":"120","added-position-deletes":"10","changed-partition-count":"3","total-records":"90","total-files-size":"4007","total-data-files":"5","total-delete-files":"3","total-position-deletes":"10","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2245381059480343696-1-23ca5b2b-d3f2-44b4-8494-f0aadd4e0b34.avro","schema-id":0,"first-row-id":90,"added-rows":0},{"sequence-number":3,"snapshot-id":7131460395349610904,"parent-snapshot-id":2245381059480343696,"timestamp-ms":1783931356769,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","removed-delete-files":"2","added-dvs":"3","removed-dvs":"2","added-files-size":"133","removed-files-size":"81","added-position-deletes":"15","removed-position-deletes":"5","changed-partition-count":"3","total-records":"90","total-files-size":"4059","total-data-files":"5","total-delete-files":"4","total-position-deletes":"20","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-7131460395349610904-1-321000ac-b08e-430f-9c79-e40d5e4aa42d.avro","schema-id":0,"first-row-id":90,"added-rows":0},{"sequence-number":4,"snapshot-id":243132775900853210,"parent-snapshot-id":7131460395349610904,"timestamp-ms":1783931357067,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"100","added-files-size":"3936","changed-partition-count":"5","total-records":"190","total-files-size":"7995","total-data-files":"10","total-delete-files":"4","total-position-deletes":"20","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-243132775900853210-1-0c03abfe-66a8-43ab-a765-a9e8072b59ed.avro","schema-id":0,"first-row-id":90,"added-rows":100},{"sequence-number":5,"snapshot-id":212327459647764229,"parent-snapshot-id":243132775900853210,"timestamp-ms":1783931357425,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"5","added-dvs":"5","added-files-size":"195","added-position-deletes":"50","changed-partition-count":"5","total-records":"190","total-files-size":"8190","total-data-files":"10","total-delete-files":"9","total-position-deletes":"70","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-212327459647764229-1-c21c4572-6006-4649-85b1-47e82e73a396.avro","schema-id":0,"first-row-id":190,"added-rows":0}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1783931356056,"snapshot-id":8352556540874311077},{"timestamp-ms":1783931356399,"snapshot-id":2245381059480343696},{"timestamp-ms":1783931356769,"snapshot-id":7131460395349610904},{"timestamp-ms":1783931357067,"snapshot-id":243132775900853210},{"timestamp-ms":1783931357425,"snapshot-id":212327459647764229}],"metadata-log":[{"timestamp-ms":1783931355607,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v1.metadata.json"},{"timestamp-ms":1783931356056,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v2.metadata.json"},{"timestamp-ms":1783931356399,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v3.metadata.json"},{"timestamp-ms":1783931356769,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v4.metadata.json"},{"timestamp-ms":1783931357067,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v5.metadata.json"}]} \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v7.metadata.json b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v7.metadata.json new file mode 100644 index 000000000000..83d3f5ec91f6 --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v7.metadata.json @@ -0,0 +1 @@ +{"format-version":3,"table-uuid":"00ae7d2e-f37d-4599-83fc-756434b2c1fe","location":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex","last-sequence-number":5,"last-updated-ms":1783931357478,"last-column-id":3,"current-schema-id":1,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"}]},{"type":"struct","schema-id":1,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"},{"id":3,"name":"label","required":false,"type":"string"}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[{"name":"id_bucket","transform":"bucket[5]","source-id":1,"field-id":1000}]}],"last-partition-id":1000,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"owner":"iantonspb","write.merge.mode":"merge-on-read","write.update.mode":"merge-on-read","write.delete.mode":"merge-on-read","write.parquet.compression-codec":"zstd"},"current-snapshot-id":212327459647764229,"next-row-id":190,"refs":{"main":{"snapshot-id":212327459647764229,"type":"branch"}},"snapshots":[{"sequence-number":1,"snapshot-id":8352556540874311077,"timestamp-ms":1783931356056,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"90","added-files-size":"3887","changed-partition-count":"5","total-records":"90","total-files-size":"3887","total-data-files":"5","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8352556540874311077-1-39ec1ceb-2dbe-4ab7-ab4e-0d18bed2d2da.avro","schema-id":0,"first-row-id":0,"added-rows":90},{"sequence-number":2,"snapshot-id":2245381059480343696,"parent-snapshot-id":8352556540874311077,"timestamp-ms":1783931356399,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","added-dvs":"3","added-files-size":"120","added-position-deletes":"10","changed-partition-count":"3","total-records":"90","total-files-size":"4007","total-data-files":"5","total-delete-files":"3","total-position-deletes":"10","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2245381059480343696-1-23ca5b2b-d3f2-44b4-8494-f0aadd4e0b34.avro","schema-id":0,"first-row-id":90,"added-rows":0},{"sequence-number":3,"snapshot-id":7131460395349610904,"parent-snapshot-id":2245381059480343696,"timestamp-ms":1783931356769,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","removed-delete-files":"2","added-dvs":"3","removed-dvs":"2","added-files-size":"133","removed-files-size":"81","added-position-deletes":"15","removed-position-deletes":"5","changed-partition-count":"3","total-records":"90","total-files-size":"4059","total-data-files":"5","total-delete-files":"4","total-position-deletes":"20","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-7131460395349610904-1-321000ac-b08e-430f-9c79-e40d5e4aa42d.avro","schema-id":0,"first-row-id":90,"added-rows":0},{"sequence-number":4,"snapshot-id":243132775900853210,"parent-snapshot-id":7131460395349610904,"timestamp-ms":1783931357067,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"100","added-files-size":"3936","changed-partition-count":"5","total-records":"190","total-files-size":"7995","total-data-files":"10","total-delete-files":"4","total-position-deletes":"20","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-243132775900853210-1-0c03abfe-66a8-43ab-a765-a9e8072b59ed.avro","schema-id":0,"first-row-id":90,"added-rows":100},{"sequence-number":5,"snapshot-id":212327459647764229,"parent-snapshot-id":243132775900853210,"timestamp-ms":1783931357425,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"5","added-dvs":"5","added-files-size":"195","added-position-deletes":"50","changed-partition-count":"5","total-records":"190","total-files-size":"8190","total-data-files":"10","total-delete-files":"9","total-position-deletes":"70","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-212327459647764229-1-c21c4572-6006-4649-85b1-47e82e73a396.avro","schema-id":0,"first-row-id":190,"added-rows":0}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1783931356056,"snapshot-id":8352556540874311077},{"timestamp-ms":1783931356399,"snapshot-id":2245381059480343696},{"timestamp-ms":1783931356769,"snapshot-id":7131460395349610904},{"timestamp-ms":1783931357067,"snapshot-id":243132775900853210},{"timestamp-ms":1783931357425,"snapshot-id":212327459647764229}],"metadata-log":[{"timestamp-ms":1783931355607,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v1.metadata.json"},{"timestamp-ms":1783931356056,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v2.metadata.json"},{"timestamp-ms":1783931356399,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v3.metadata.json"},{"timestamp-ms":1783931356769,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v4.metadata.json"},{"timestamp-ms":1783931357067,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v5.metadata.json"},{"timestamp-ms":1783931357425,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v6.metadata.json"}]} \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v8.metadata.json b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v8.metadata.json new file mode 100644 index 000000000000..3479448d1bf0 --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v8.metadata.json @@ -0,0 +1 @@ +{"format-version":3,"table-uuid":"00ae7d2e-f37d-4599-83fc-756434b2c1fe","location":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex","last-sequence-number":6,"last-updated-ms":1783931357776,"last-column-id":3,"current-schema-id":1,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"}]},{"type":"struct","schema-id":1,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"},{"id":3,"name":"label","required":false,"type":"string"}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[{"name":"id_bucket","transform":"bucket[5]","source-id":1,"field-id":1000}]}],"last-partition-id":1000,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"owner":"iantonspb","write.merge.mode":"merge-on-read","write.update.mode":"merge-on-read","write.delete.mode":"merge-on-read","write.parquet.compression-codec":"zstd"},"current-snapshot-id":2216958009875447676,"next-row-id":240,"refs":{"main":{"snapshot-id":2216958009875447676,"type":"branch"}},"snapshots":[{"sequence-number":1,"snapshot-id":8352556540874311077,"timestamp-ms":1783931356056,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"90","added-files-size":"3887","changed-partition-count":"5","total-records":"90","total-files-size":"3887","total-data-files":"5","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8352556540874311077-1-39ec1ceb-2dbe-4ab7-ab4e-0d18bed2d2da.avro","schema-id":0,"first-row-id":0,"added-rows":90},{"sequence-number":2,"snapshot-id":2245381059480343696,"parent-snapshot-id":8352556540874311077,"timestamp-ms":1783931356399,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","added-dvs":"3","added-files-size":"120","added-position-deletes":"10","changed-partition-count":"3","total-records":"90","total-files-size":"4007","total-data-files":"5","total-delete-files":"3","total-position-deletes":"10","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2245381059480343696-1-23ca5b2b-d3f2-44b4-8494-f0aadd4e0b34.avro","schema-id":0,"first-row-id":90,"added-rows":0},{"sequence-number":3,"snapshot-id":7131460395349610904,"parent-snapshot-id":2245381059480343696,"timestamp-ms":1783931356769,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","removed-delete-files":"2","added-dvs":"3","removed-dvs":"2","added-files-size":"133","removed-files-size":"81","added-position-deletes":"15","removed-position-deletes":"5","changed-partition-count":"3","total-records":"90","total-files-size":"4059","total-data-files":"5","total-delete-files":"4","total-position-deletes":"20","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-7131460395349610904-1-321000ac-b08e-430f-9c79-e40d5e4aa42d.avro","schema-id":0,"first-row-id":90,"added-rows":0},{"sequence-number":4,"snapshot-id":243132775900853210,"parent-snapshot-id":7131460395349610904,"timestamp-ms":1783931357067,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"100","added-files-size":"3936","changed-partition-count":"5","total-records":"190","total-files-size":"7995","total-data-files":"10","total-delete-files":"4","total-position-deletes":"20","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-243132775900853210-1-0c03abfe-66a8-43ab-a765-a9e8072b59ed.avro","schema-id":0,"first-row-id":90,"added-rows":100},{"sequence-number":5,"snapshot-id":212327459647764229,"parent-snapshot-id":243132775900853210,"timestamp-ms":1783931357425,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"5","added-dvs":"5","added-files-size":"195","added-position-deletes":"50","changed-partition-count":"5","total-records":"190","total-files-size":"8190","total-data-files":"10","total-delete-files":"9","total-position-deletes":"70","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-212327459647764229-1-c21c4572-6006-4649-85b1-47e82e73a396.avro","schema-id":0,"first-row-id":190,"added-rows":0},{"sequence-number":6,"snapshot-id":2216958009875447676,"parent-snapshot-id":212327459647764229,"timestamp-ms":1783931357776,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"50","added-files-size":"5103","changed-partition-count":"5","total-records":"240","total-files-size":"13293","total-data-files":"15","total-delete-files":"9","total-position-deletes":"70","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2216958009875447676-1-b68e784a-6b9c-4894-827d-c56978599ecd.avro","schema-id":1,"first-row-id":190,"added-rows":50}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1783931356056,"snapshot-id":8352556540874311077},{"timestamp-ms":1783931356399,"snapshot-id":2245381059480343696},{"timestamp-ms":1783931356769,"snapshot-id":7131460395349610904},{"timestamp-ms":1783931357067,"snapshot-id":243132775900853210},{"timestamp-ms":1783931357425,"snapshot-id":212327459647764229},{"timestamp-ms":1783931357776,"snapshot-id":2216958009875447676}],"metadata-log":[{"timestamp-ms":1783931355607,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v1.metadata.json"},{"timestamp-ms":1783931356056,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v2.metadata.json"},{"timestamp-ms":1783931356399,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v3.metadata.json"},{"timestamp-ms":1783931356769,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v4.metadata.json"},{"timestamp-ms":1783931357067,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v5.metadata.json"},{"timestamp-ms":1783931357425,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v6.metadata.json"},{"timestamp-ms":1783931357478,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v7.metadata.json"}]} \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v9.metadata.json b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v9.metadata.json new file mode 100644 index 000000000000..c317c0743b1b --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v9.metadata.json @@ -0,0 +1 @@ +{"format-version":3,"table-uuid":"00ae7d2e-f37d-4599-83fc-756434b2c1fe","location":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex","last-sequence-number":7,"last-updated-ms":1783931358118,"last-column-id":3,"current-schema-id":1,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"}]},{"type":"struct","schema-id":1,"fields":[{"id":1,"name":"id","required":false,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"},{"id":3,"name":"label","required":false,"type":"string"}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[{"name":"id_bucket","transform":"bucket[5]","source-id":1,"field-id":1000}]}],"last-partition-id":1000,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"owner":"iantonspb","write.merge.mode":"merge-on-read","write.update.mode":"merge-on-read","write.delete.mode":"merge-on-read","write.parquet.compression-codec":"zstd"},"current-snapshot-id":399037466005519109,"next-row-id":240,"refs":{"main":{"snapshot-id":399037466005519109,"type":"branch"}},"snapshots":[{"sequence-number":1,"snapshot-id":8352556540874311077,"timestamp-ms":1783931356056,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"90","added-files-size":"3887","changed-partition-count":"5","total-records":"90","total-files-size":"3887","total-data-files":"5","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-8352556540874311077-1-39ec1ceb-2dbe-4ab7-ab4e-0d18bed2d2da.avro","schema-id":0,"first-row-id":0,"added-rows":90},{"sequence-number":2,"snapshot-id":2245381059480343696,"parent-snapshot-id":8352556540874311077,"timestamp-ms":1783931356399,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","added-dvs":"3","added-files-size":"120","added-position-deletes":"10","changed-partition-count":"3","total-records":"90","total-files-size":"4007","total-data-files":"5","total-delete-files":"3","total-position-deletes":"10","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2245381059480343696-1-23ca5b2b-d3f2-44b4-8494-f0aadd4e0b34.avro","schema-id":0,"first-row-id":90,"added-rows":0},{"sequence-number":3,"snapshot-id":7131460395349610904,"parent-snapshot-id":2245381059480343696,"timestamp-ms":1783931356769,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","removed-delete-files":"2","added-dvs":"3","removed-dvs":"2","added-files-size":"133","removed-files-size":"81","added-position-deletes":"15","removed-position-deletes":"5","changed-partition-count":"3","total-records":"90","total-files-size":"4059","total-data-files":"5","total-delete-files":"4","total-position-deletes":"20","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-7131460395349610904-1-321000ac-b08e-430f-9c79-e40d5e4aa42d.avro","schema-id":0,"first-row-id":90,"added-rows":0},{"sequence-number":4,"snapshot-id":243132775900853210,"parent-snapshot-id":7131460395349610904,"timestamp-ms":1783931357067,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"100","added-files-size":"3936","changed-partition-count":"5","total-records":"190","total-files-size":"7995","total-data-files":"10","total-delete-files":"4","total-position-deletes":"20","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-243132775900853210-1-0c03abfe-66a8-43ab-a765-a9e8072b59ed.avro","schema-id":0,"first-row-id":90,"added-rows":100},{"sequence-number":5,"snapshot-id":212327459647764229,"parent-snapshot-id":243132775900853210,"timestamp-ms":1783931357425,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"5","added-dvs":"5","added-files-size":"195","added-position-deletes":"50","changed-partition-count":"5","total-records":"190","total-files-size":"8190","total-data-files":"10","total-delete-files":"9","total-position-deletes":"70","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-212327459647764229-1-c21c4572-6006-4649-85b1-47e82e73a396.avro","schema-id":0,"first-row-id":190,"added-rows":0},{"sequence-number":6,"snapshot-id":2216958009875447676,"parent-snapshot-id":212327459647764229,"timestamp-ms":1783931357776,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"5","added-records":"50","added-files-size":"5103","changed-partition-count":"5","total-records":"240","total-files-size":"13293","total-data-files":"15","total-delete-files":"9","total-position-deletes":"70","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-2216958009875447676-1-b68e784a-6b9c-4894-827d-c56978599ecd.avro","schema-id":1,"first-row-id":190,"added-rows":50},{"sequence-number":7,"snapshot-id":399037466005519109,"parent-snapshot-id":2216958009875447676,"timestamp-ms":1783931358118,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"3","added-dvs":"3","added-files-size":"126","added-position-deletes":"3","changed-partition-count":"3","total-records":"240","total-files-size":"13419","total-data-files":"15","total-delete-files":"12","total-position-deletes":"73","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/snap-399037466005519109-1-6a19b33f-2be5-4073-820c-82cb1c7d76cb.avro","schema-id":1,"first-row-id":240,"added-rows":0}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1783931356056,"snapshot-id":8352556540874311077},{"timestamp-ms":1783931356399,"snapshot-id":2245381059480343696},{"timestamp-ms":1783931356769,"snapshot-id":7131460395349610904},{"timestamp-ms":1783931357067,"snapshot-id":243132775900853210},{"timestamp-ms":1783931357425,"snapshot-id":212327459647764229},{"timestamp-ms":1783931357776,"snapshot-id":2216958009875447676},{"timestamp-ms":1783931358118,"snapshot-id":399037466005519109}],"metadata-log":[{"timestamp-ms":1783931355607,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v1.metadata.json"},{"timestamp-ms":1783931356056,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v2.metadata.json"},{"timestamp-ms":1783931356399,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v3.metadata.json"},{"timestamp-ms":1783931356769,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v4.metadata.json"},{"timestamp-ms":1783931357067,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v5.metadata.json"},{"timestamp-ms":1783931357425,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v6.metadata.json"},{"timestamp-ms":1783931357478,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v7.metadata.json"},{"timestamp-ms":1783931357776,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/v8.metadata.json"}]} \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/version-hint.text b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/version-hint.text new file mode 100644 index 000000000000..9d607966b721 --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_complex/metadata/version-hint.text @@ -0,0 +1 @@ +11 \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/data/00000-0-e150e3cb-5697-43f9-b2ea-9894139e9373-0-00001.parquet b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/data/00000-0-e150e3cb-5697-43f9-b2ea-9894139e9373-0-00001.parquet new file mode 100644 index 000000000000..4a32f9743fe7 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/data/00000-0-e150e3cb-5697-43f9-b2ea-9894139e9373-0-00001.parquet differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/data/00000-2-ee31c25d-1223-465e-b4d7-6366f5ebda40-00001-deletes.puffin b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/data/00000-2-ee31c25d-1223-465e-b4d7-6366f5ebda40-00001-deletes.puffin new file mode 100644 index 000000000000..64c688f02ccd Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/data/00000-2-ee31c25d-1223-465e-b4d7-6366f5ebda40-00001-deletes.puffin differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/0e8edb12-fdf0-4ce1-b6ab-c9c2a05d6d57-m0.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/0e8edb12-fdf0-4ce1-b6ab-c9c2a05d6d57-m0.avro new file mode 100644 index 000000000000..1f1d617f028d Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/0e8edb12-fdf0-4ce1-b6ab-c9c2a05d6d57-m0.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/c34940c4-6b84-4560-85a3-35e17802d85d-m0.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/c34940c4-6b84-4560-85a3-35e17802d85d-m0.avro new file mode 100644 index 000000000000..684c55f993d7 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/c34940c4-6b84-4560-85a3-35e17802d85d-m0.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/snap-1455293623190430422-1-c34940c4-6b84-4560-85a3-35e17802d85d.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/snap-1455293623190430422-1-c34940c4-6b84-4560-85a3-35e17802d85d.avro new file mode 100644 index 000000000000..a968682b674d Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/snap-1455293623190430422-1-c34940c4-6b84-4560-85a3-35e17802d85d.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/snap-8278841256334127309-1-0e8edb12-fdf0-4ce1-b6ab-c9c2a05d6d57.avro b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/snap-8278841256334127309-1-0e8edb12-fdf0-4ce1-b6ab-c9c2a05d6d57.avro new file mode 100644 index 000000000000..ef9e9c5fc2f3 Binary files /dev/null and b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/snap-8278841256334127309-1-0e8edb12-fdf0-4ce1-b6ab-c9c2a05d6d57.avro differ diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/v1.metadata.json b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/v1.metadata.json new file mode 100644 index 000000000000..761494ac6f45 --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/v1.metadata.json @@ -0,0 +1 @@ +{"format-version":3,"table-uuid":"1d4ead84-5f74-4062-a8d2-46b419a5650d","location":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source","last-sequence-number":0,"last-updated-ms":1783931352802,"last-column-id":1,"current-schema-id":0,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[]}],"last-partition-id":999,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"owner":"iantonspb","write.merge.mode":"merge-on-read","write.update.mode":"merge-on-read","write.delete.mode":"merge-on-read","write.parquet.compression-codec":"zstd"},"current-snapshot-id":null,"next-row-id":0,"refs":{},"snapshots":[],"statistics":[],"partition-statistics":[],"snapshot-log":[],"metadata-log":[]} \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/v2.metadata.json b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/v2.metadata.json new file mode 100644 index 000000000000..6470e30e9314 --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/v2.metadata.json @@ -0,0 +1 @@ +{"format-version":3,"table-uuid":"1d4ead84-5f74-4062-a8d2-46b419a5650d","location":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source","last-sequence-number":1,"last-updated-ms":1783931354428,"last-column-id":1,"current-schema-id":0,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[]}],"last-partition-id":999,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"owner":"iantonspb","write.merge.mode":"merge-on-read","write.update.mode":"merge-on-read","write.delete.mode":"merge-on-read","write.parquet.compression-codec":"zstd"},"current-snapshot-id":1455293623190430422,"next-row-id":200,"refs":{"main":{"snapshot-id":1455293623190430422,"type":"branch"}},"snapshots":[{"sequence-number":1,"snapshot-id":1455293623190430422,"timestamp-ms":1783931354428,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"1","added-records":"200","added-files-size":"734","changed-partition-count":"1","total-records":"200","total-files-size":"734","total-data-files":"1","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/snap-1455293623190430422-1-c34940c4-6b84-4560-85a3-35e17802d85d.avro","schema-id":0,"first-row-id":0,"added-rows":200}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1783931354428,"snapshot-id":1455293623190430422}],"metadata-log":[{"timestamp-ms":1783931352802,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/v1.metadata.json"}]} \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/v3.metadata.json b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/v3.metadata.json new file mode 100644 index 000000000000..d3e24bdc9b16 --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/v3.metadata.json @@ -0,0 +1 @@ +{"format-version":3,"table-uuid":"1d4ead84-5f74-4062-a8d2-46b419a5650d","location":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source","last-sequence-number":2,"last-updated-ms":1783931355547,"last-column-id":1,"current-schema-id":0,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long"}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[]}],"last-partition-id":999,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"owner":"iantonspb","write.merge.mode":"merge-on-read","write.update.mode":"merge-on-read","write.delete.mode":"merge-on-read","write.parquet.compression-codec":"zstd"},"current-snapshot-id":8278841256334127309,"next-row-id":200,"refs":{"main":{"snapshot-id":8278841256334127309,"type":"branch"}},"snapshots":[{"sequence-number":1,"snapshot-id":1455293623190430422,"timestamp-ms":1783931354428,"summary":{"operation":"append","spark.app.id":"local-1783931350618","added-data-files":"1","added-records":"200","added-files-size":"734","changed-partition-count":"1","total-records":"200","total-files-size":"734","total-data-files":"1","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/snap-1455293623190430422-1-c34940c4-6b84-4560-85a3-35e17802d85d.avro","schema-id":0,"first-row-id":0,"added-rows":200},{"sequence-number":2,"snapshot-id":8278841256334127309,"parent-snapshot-id":1455293623190430422,"timestamp-ms":1783931355547,"summary":{"operation":"delete","spark.app.id":"local-1783931350618","added-delete-files":"1","added-dvs":"1","added-files-size":"48","added-position-deletes":"4","changed-partition-count":"1","total-records":"200","total-files-size":"782","total-data-files":"1","total-delete-files":"1","total-position-deletes":"4","total-equality-deletes":"0","engine-version":"3.5.3","app-id":"local-1783931350618","engine-name":"spark","iceberg-version":"Apache Iceberg unspecified (commit 7dbafb438ee1e68d0047bebcb587265d7d87d8a1)"},"manifest-list":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/snap-8278841256334127309-1-0e8edb12-fdf0-4ce1-b6ab-c9c2a05d6d57.avro","schema-id":0,"first-row-id":200,"added-rows":0}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1783931354428,"snapshot-id":1455293623190430422},{"timestamp-ms":1783931355547,"snapshot-id":8278841256334127309}],"metadata-log":[{"timestamp-ms":1783931352802,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/v1.metadata.json"},{"timestamp-ms":1783931354428,"metadata-file":"/home/iantonspb/ClickHouse/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/v2.metadata.json"}]} \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/version-hint.text b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/version-hint.text new file mode 100644 index 000000000000..e440e5c84258 --- /dev/null +++ b/tests/queries/0_stateless/data_minio/dv_puffin_warehouse/default/dv_puffin_source/metadata/version-hint.text @@ -0,0 +1 @@ +3 \ No newline at end of file diff --git a/tests/queries/0_stateless/data_minio/generate_iceberg_dv_fixture.py b/tests/queries/0_stateless/data_minio/generate_iceberg_dv_fixture.py new file mode 100644 index 000000000000..86e4ffd08804 --- /dev/null +++ b/tests/queries/0_stateless/data_minio/generate_iceberg_dv_fixture.py @@ -0,0 +1,160 @@ +#!/usr/bin/env python3 +"""Generate Iceberg v3 table fixtures with Puffin deletion vectors for ClickHouse tests. + +Requirements: + - Java 11+ + - pyspark + - Network access on first run (downloads iceberg-spark-runtime) + +Usage: + python3 generate_iceberg_dv_fixture.py [warehouse_dir] + python3 generate_iceberg_dv_fixture.py --complex-only [warehouse_dir] +""" + +from __future__ import annotations + +import shutil +import sys +from pathlib import Path + +try: + from pyspark.sql import SparkSession +except ImportError as exc: # pragma: no cover - helper script + raise SystemExit("pyspark is required") from exc + +DEFAULT_WAREHOUSE = Path(__file__).resolve().parent / "dv_puffin_warehouse" +ICEBERG_PACKAGE = "org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.9.0" +SIMPLE_TABLE_NAME = "dv_puffin_source" +COMPLEX_TABLE_NAME = "dv_puffin_complex" +SIMPLE_DELETED_IDS = [2, 5, 7, 100] +COMPLEX_DELETED_IDS = [205, 210, 220] + + +def build_spark_session(warehouse: Path) -> SparkSession: + return ( + SparkSession.builder.appName("generate_iceberg_dv_fixture") + .master("local[1]") + .config("spark.jars.packages", ICEBERG_PACKAGE) + .config( + "spark.sql.extensions", + "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions", + ) + .config("spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkSessionCatalog") + .config("spark.sql.catalog.spark_catalog.type", "hadoop") + .config("spark.sql.catalog.spark_catalog.warehouse", str(warehouse)) + .config("spark.ui.enabled", "false") + .getOrCreate() + ) + + +def generate_simple_fixture(spark: SparkSession) -> None: + spark.sql( + f""" + CREATE TABLE default.{SIMPLE_TABLE_NAME} (id BIGINT) + USING iceberg + TBLPROPERTIES ( + 'format-version' = '3', + 'write.delete.mode' = 'merge-on-read', + 'write.update.mode' = 'merge-on-read', + 'write.merge.mode' = 'merge-on-read' + ) + """ + ) + spark.sql(f"INSERT INTO default.{SIMPLE_TABLE_NAME} SELECT id FROM range(0, 200)") + spark.sql( + f"DELETE FROM default.{SIMPLE_TABLE_NAME} " + f"WHERE id IN ({', '.join(str(x) for x in SIMPLE_DELETED_IDS)})" + ) + + +def generate_complex_fixture(spark: SparkSession) -> None: + spark.sql( + f""" + CREATE TABLE default.{COMPLEX_TABLE_NAME} (id BIGINT, data STRING) + USING iceberg + PARTITIONED BY (bucket(5, id)) + TBLPROPERTIES ( + 'format-version' = '3', + 'write.delete.mode' = 'merge-on-read', + 'write.update.mode' = 'merge-on-read', + 'write.merge.mode' = 'merge-on-read' + ) + """ + ) + spark.sql( + f"INSERT INTO default.{COMPLEX_TABLE_NAME} " + f"SELECT id, char(id + ascii('a')) FROM range(10, 100)" + ) + spark.sql(f"DELETE FROM default.{COMPLEX_TABLE_NAME} WHERE id < 20") + spark.sql(f"DELETE FROM default.{COMPLEX_TABLE_NAME} WHERE id >= 90") + spark.sql( + f"INSERT INTO default.{COMPLEX_TABLE_NAME} " + f"SELECT id, char(id + ascii('a')) FROM range(100, 200)" + ) + spark.sql(f"DELETE FROM default.{COMPLEX_TABLE_NAME} WHERE id >= 150") + spark.sql(f"ALTER TABLE default.{COMPLEX_TABLE_NAME} ADD COLUMNS (label STRING)") + spark.sql( + f""" + INSERT INTO default.{COMPLEX_TABLE_NAME} + SELECT id, char(id + ascii('a')), 'new' + FROM range(200, 250) + """ + ) + spark.sql( + f"DELETE FROM default.{COMPLEX_TABLE_NAME} " + f"WHERE id IN ({', '.join(str(x) for x in COMPLEX_DELETED_IDS)})" + ) + spark.sql(f"UPDATE default.{COMPLEX_TABLE_NAME} SET label = 'updated' WHERE id = 25") + spark.sql(f"CALL system.rewrite_data_files(table => 'default.{COMPLEX_TABLE_NAME}')") + + +def cleanup_crc_files(table_dir: Path) -> None: + for crc_file in table_dir.rglob("*.crc"): + crc_file.unlink() + + +def generate_fixture(warehouse: Path, *, simple: bool, complex_table: bool) -> None: + spark = build_spark_session(warehouse) + try: + if simple: + generate_simple_fixture(spark) + if complex_table: + generate_complex_fixture(spark) + finally: + spark.stop() + + if simple: + simple_dir = warehouse / "default" / SIMPLE_TABLE_NAME + if not simple_dir.exists(): + raise RuntimeError(f"Expected table directory at {simple_dir}") + cleanup_crc_files(simple_dir) + + if complex_table: + complex_dir = warehouse / "default" / COMPLEX_TABLE_NAME + if not complex_dir.exists(): + raise RuntimeError(f"Expected table directory at {complex_dir}") + cleanup_crc_files(complex_dir) + + +def main() -> None: + args = sys.argv[1:] + complex_only = False + if args and args[0] == "--complex-only": + complex_only = True + args = args[1:] + + warehouse = Path(args[0]).resolve() if args else DEFAULT_WAREHOUSE.resolve() + if warehouse.exists(): + shutil.rmtree(warehouse) + warehouse.mkdir(parents=True) + + generate_fixture(warehouse, simple=not complex_only, complex_table=True) + + if complex_only: + print(f"Wrote complex Iceberg v3 deletion vector fixture to {warehouse / 'default' / COMPLEX_TABLE_NAME}") + else: + print(f"Wrote Iceberg v3 deletion vector fixtures to {warehouse / 'default'}") + + +if __name__ == "__main__": + main() diff --git a/tests/queries/0_stateless/data_puffin/dv_sequence_number_not_minus_one.puffin b/tests/queries/0_stateless/data_puffin/dv_sequence_number_not_minus_one.puffin new file mode 100644 index 000000000000..9b260e2de1d5 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/dv_sequence_number_not_minus_one.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/dv_snapshot_id_not_minus_one.puffin b/tests/queries/0_stateless/data_puffin/dv_snapshot_id_not_minus_one.puffin new file mode 100644 index 000000000000..7216285d651e Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/dv_snapshot_id_not_minus_one.puffin differ diff --git a/tests/queries/0_stateless/data_puffin/generate_puffin_fixtures.py b/tests/queries/0_stateless/data_puffin/generate_puffin_fixtures.py index dac6eb811559..120643cf21ad 100644 --- a/tests/queries/0_stateless/data_puffin/generate_puffin_fixtures.py +++ b/tests/queries/0_stateless/data_puffin/generate_puffin_fixtures.py @@ -5,24 +5,11 @@ import json import struct -import subprocess -import sys import zlib from pathlib import Path - -try: - import lz4.frame - import xxhash -except ImportError: - subprocess.check_call([sys.executable, "-m", "pip", "install", "lz4", "xxhash", "-q"]) - import lz4.frame - import xxhash - -try: - import pyroaring -except ImportError: - subprocess.check_call([sys.executable, "-m", "pip", "install", "pyroaring", "-q"]) - import pyroaring +import lz4.frame +import xxhash +import pyroaring OUTPUT_DIR = Path(__file__).parent PUFFIN_MAGIC = b"PFA1" @@ -345,6 +332,21 @@ def generate_missing_required_fields() -> None: ), ) + # Puffin v1 DV blobs must keep snapshot-id / sequence-number at -1. + for name, field, value in ( + ("dv_snapshot_id_not_minus_one.puffin", "snapshot-id", 0), + ("dv_sequence_number_not_minus_one.puffin", "sequence-number", 0), + ): + case_payload = json.loads(footer_json.decode("utf-8")) + case_payload["blobs"][0][field] = value + write_fixture( + name, + build_puffin_file( + BLOB_PLACEHOLDER, + json.dumps(case_payload, separators=(", ", ": ")).encode("utf-8"), + ), + ) + def generate_invalid_property_value_types() -> None: """Property maps must have string values; non-strings must fail with BAD_ARGUMENTS.""" @@ -618,6 +620,48 @@ def generate_invalid_file_metadata_properties() -> None: ) +def generate_mixed_blob_types() -> None: + """DV plus a non-DV blob: footer parse and Puffin SQL must tolerate the sketch entry.""" + ok = OUTPUT_DIR / "file_properties_ok.puffin" + if not ok.exists(): + raise SystemExit("file_properties_ok.puffin required to build mixed_blob_types.puffin") + + sketch = b"\x00" * 16 + puffin = ok.read_bytes() + blob_end = puffin.index(PUFFIN_MAGIC, 4) + dv = puffin[4:blob_end] + + footer = { + "blobs": [ + { + "type": "apache-datasketches-theta-v1", + "fields": [], + "snapshot-id": -1, + "sequence-number": -1, + "offset": 4, + "length": len(sketch), + "properties": {}, + }, + { + "type": "deletion-vector-v1", + "fields": [], + "snapshot-id": -1, + "sequence-number": -1, + "offset": 4 + len(sketch), + "length": len(dv), + "properties": default_dv_properties(cardinality="2"), + }, + ] + } + write_fixture( + "mixed_blob_types.puffin", + build_puffin_file_from_blobs( + [sketch, dv], + json.dumps(footer, separators=(", ", ": ")).encode("utf-8"), + ), + ) + + def generate_unparseable_footer_json() -> None: """Malformed JSON / oversize integers must fail with BAD_ARGUMENTS, not STD_EXCEPTION.""" write_raw_footer_fixture("malformed_footer_json.puffin", b"{") @@ -653,6 +697,7 @@ def main() -> None: generate_missing_footer_leading_magic() generate_invalid_file_metadata_properties() generate_unparseable_footer_json() + generate_mixed_blob_types() generate_cardinality_mismatch_large_bitmap() generate_dense_range_100k() diff --git a/tests/queries/0_stateless/data_puffin/mixed_blob_types.puffin b/tests/queries/0_stateless/data_puffin/mixed_blob_types.puffin new file mode 100644 index 000000000000..90c0f6dccb26 Binary files /dev/null and b/tests/queries/0_stateless/data_puffin/mixed_blob_types.puffin differ