Antalya 26.6: Iceberg: support external paths in tables - #2154
Conversation
…olution in next commit)
Kept the antalya-26.6 side for everything outside the source PR's scope (older cluster-protocol version list, non-lambda iceberg_metadata_log->add, 4-arg createReadBuffer, string-path tryGetObjectMetadata/getObjectMetadata, try/catch around generateManifestList in Mutations, no manifest-only compaction) and applied the PR's changes on top of it. Also removed the upstream 26.7/26.8 SettingsChangesHistory blocks that the three-way merge re-introduced into antalya-26.6's registry, and uncommented the existing Antalya placeholder row for the PR's new setting instead of adding a duplicate. Adapted: ObjectInfo::getIdentifierForPath(path) added on antalya-26.6 (getIdentifier() refactored to use it) because the PR's getSchedulingIdentifier() relies on that accessor, which only exists upstream Adapted: createReadBuffer / tryGetObjectMetadata / getObjectMetadata call sites keep antalya-26.6's argument shapes (string path, no read_settings/headers args) and only swap the object storage for the resolved one Adapted: generateManifestList call sites keep antalya-26.6's signature (per-call content_type, no per_entry_content_types) and only pass the new secondary_storages argument Adapted: manifest_file_cache_keys.emplace_back keeps antalya-26.6's 5-field ManifestFileCacheKey (no partition_spec_id) Adapted: Iceberg/Utils.h needs an explicit #include <optional> for the new always-available declarations, since antalya-26.6 had it only inside the USE_AVRO block Dropped: manifest-only compaction integration (compactIcebergManifests, writeConsolidatedManifestFile, isCurrentManifestListAboveThreshold, IcebergMetadata::optimizeManifestFiles) — that feature is not on antalya-26.6, only the PR's edits to it were dropped Dropped: query-condition-cache keying by the Iceberg metadata path (makeQueryConditionCacheKey, ObjectInfo::getIdentifier(bool)/getIdentifierForPath(path, bool) overloads) — depends on upstream query-condition-cache/ETag work not on antalya-26.6 Dropped: gtest_storage_object_storage_archive.cpp update — that test file does not exist on antalya-26.6
…ouse-pr-90740 # Conflicts: # src/Storages/ObjectStorage/DataLakes/Iceberg/Utils.h
This comment was marked as outdated.
This comment was marked as outdated.
…ature/antalya-26.6/ClickHouse-ClickHouse-pr-90740
…ouse-pr-90740 # Conflicts: # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp
…ouse-pr-90740 # Conflicts: # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.cpp # src/Storages/ObjectStorage/StorageObjectStorageSource.cpp # src/Storages/ObjectStorage/StorageObjectStorageStableTaskDistributor.cpp
CI triage — 3 distinct red checks: 2 PR-caused, 1 flaky/unrelatedI looked at every failing check on
The good news first: the reader side this PR actually targets is healthy. On the base branch 🔴 1.
|
RelEasy
|
CI triage — none of the failures are caused by this PRThere are two contradicting triage comments above. I re-derived the answer from the raw testflows artifacts rather than from either summary. Result: all failing checks are pre-existing on Why the export suites suddenly went redBoth export suites are gated behind a build-feature probe. On every recent base-branch
The gate is That feature reached Control PRs on the same merge baseComparing sorted leaf-
Byte-identical failure sets on both
|
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Support Iceberg tables that have files outside table location or on different storage (ClickHouse#90740 by @zvonand).
Cherry-picked from ClickHouse#90740.
Closes ClickHouse#84609
Current logic is that all table files (data files, manifests, manifest lists) shall be "inside" the table location, in the same storage. This PR abandons that logic: now files can be located anywhere, even on a different storage type (e.g. all metadata is on s3, data file in in local storage).
In some cases, that old logic shoots back: even incorrect (non-existing) paths, i.e. with redundant items in a prefix were parsed in such a way that files from another locations were read (
/path/to/itemwas read when the specified location was/path/to/to/itemor vice versa)Documentation entry for user-facing changes
CI/CD Options
Exclude tests:
Regression jobs to run: