Skip to content

Antalya 26.6: Iceberg: support external paths in tables - #2154

Open
zvonand wants to merge 8 commits into
antalya-26.6from
feature/antalya-26.6/ClickHouse-ClickHouse-pr-90740
Open

Antalya 26.6: Iceberg: support external paths in tables#2154
zvonand wants to merge 8 commits into
antalya-26.6from
feature/antalya-26.6/ClickHouse-ClickHouse-pr-90740

Conversation

@zvonand

@zvonand zvonand commented Aug 3, 2026

Copy link
Copy Markdown
Member

Dropped from this backport: the AI dropped these surfaces rather than pulling in a missing prerequisite. Reviewers: confirm each is genuinely optional.

  • manifest-only compaction integration (compactIcebergManifests, writeConsolidatedManifestFile, isCurrentManifestListAboveThreshold, IcebergMetadata::optimizeManifestFiles) — that feature is not on antalya-26.6, only the PR's edits to it were dropped
  • query-condition-cache keying by the Iceberg metadata path (makeQueryConditionCacheKey, ObjectInfo::getIdentifier(bool)/getIdentifierForPath(path, bool) overloads) — depends on upstream query-condition-cache/ETag work not on antalya-26.6
  • gtest_storage_object_storage_archive.cpp update — that test file does not exist on antalya-26.6

Changelog category (leave one):

  • Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Support Iceberg tables that have files outside table location or on different storage (ClickHouse#90740 by @zvonand).

Cherry-picked from ClickHouse#90740.


Closes ClickHouse#84609

Current logic is that all table files (data files, manifests, manifest lists) shall be "inside" the table location, in the same storage. This PR abandons that logic: now files can be located anywhere, even on a different storage type (e.g. all metadata is on s3, data file in in local storage).

In some cases, that old logic shoots back: even incorrect (non-existing) paths, i.e. with redundant items in a prefix were parsed in such a way that files from another locations were read (/path/to/item was read when the specified location was /path/to/to/item or vice versa)

Documentation entry for user-facing changes

  • Documentation is written (mandatory for new features)

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Performance tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All with Aarch64
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

zvonand added 2 commits August 3, 2026 18:36
Kept the antalya-26.6 side for everything outside the source PR's scope
(older cluster-protocol version list, non-lambda iceberg_metadata_log->add,
4-arg createReadBuffer, string-path tryGetObjectMetadata/getObjectMetadata,
try/catch around generateManifestList in Mutations, no manifest-only
compaction) and applied the PR's changes on top of it.

Also removed the upstream 26.7/26.8 SettingsChangesHistory blocks that the
three-way merge re-introduced into antalya-26.6's registry, and uncommented
the existing Antalya placeholder row for the PR's new setting instead of
adding a duplicate.

Adapted: ObjectInfo::getIdentifierForPath(path) added on antalya-26.6 (getIdentifier() refactored to use it) because the PR's getSchedulingIdentifier() relies on that accessor, which only exists upstream
Adapted: createReadBuffer / tryGetObjectMetadata / getObjectMetadata call sites keep antalya-26.6's argument shapes (string path, no read_settings/headers args) and only swap the object storage for the resolved one
Adapted: generateManifestList call sites keep antalya-26.6's signature (per-call content_type, no per_entry_content_types) and only pass the new secondary_storages argument
Adapted: manifest_file_cache_keys.emplace_back keeps antalya-26.6's 5-field ManifestFileCacheKey (no partition_spec_id)
Adapted: Iceberg/Utils.h needs an explicit #include <optional> for the new always-available declarations, since antalya-26.6 had it only inside the USE_AVRO block
Dropped: manifest-only compaction integration (compactIcebergManifests, writeConsolidatedManifestFile, isCurrentManifestListAboveThreshold, IcebergMetadata::optimizeManifestFiles) — that feature is not on antalya-26.6, only the PR's edits to it were dropped
Dropped: query-condition-cache keying by the Iceberg metadata path (makeQueryConditionCacheKey, ObjectInfo::getIdentifier(bool)/getIdentifierForPath(path, bool) overloads) — depends on upstream query-condition-cache/ETag work not on antalya-26.6
Dropped: gtest_storage_object_storage_archive.cpp update — that test file does not exist on antalya-26.6
@zvonand zvonand added releasy Created/managed by RelEasy antalya-26.6 ai-resolved Port conflict auto-resolved by Claude labels Aug 3, 2026
@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown

Workflow [PR], commit [a623cd1]

…ouse-pr-90740

# Conflicts:
#	src/Storages/ObjectStorage/DataLakes/Iceberg/Utils.h
@mkmkme

This comment was marked as outdated.

…ature/antalya-26.6/ClickHouse-ClickHouse-pr-90740
…ouse-pr-90740

# Conflicts:
#	src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp
…ouse-pr-90740

# Conflicts:
#	src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.cpp
#	src/Storages/ObjectStorage/StorageObjectStorageSource.cpp
#	src/Storages/ObjectStorage/StorageObjectStorageStableTaskDistributor.cpp
@zvonand

zvonand commented Aug 14, 2026

Copy link
Copy Markdown
Member Author

@blau-ai

@blau-ai

blau-ai commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

CI triage — 3 distinct red checks: 2 PR-caused, 1 flaky/unrelated

I looked at every failing check on a623cd1. They collapse into three real signals (the top-level PR check is just the rollup gate, and each failure reproduces identically on both aarch64 and x86_64, so nothing here is a runner one-off):

Check Verdict Root cause
Regression Iceberg (2) / iceberg_2 🔴 PR-caused Iceberg export/write path
Regression S3Export (part) / s3_export_part 🔴 PR-caused same write path
Stateless tests (arm_binary, parallel) 🟡 flaky / unrelated 03518_alter_logical_race

The good news first: the reader side this PR actually targets is healthy. On the base branch iceberg_2 fails 8 iceberg engine (read) scenarios; on this PR those are green (0 read failures). The intended improvement (external/relative Iceberg paths on read) appears to work. The regressions are all on the write side.


🔴 1. Iceberg (2) / iceberg_2 — PR-caused

  • PR: 242 leaf scenarios failed, and every one is under /iceberg/export partition (842 [Fail] lines incl. parent suites; the rest of the module passes). Fails on both arches identically.
  • Base branch (antalya-26.6, latest MasterCI 31713806632): only 14 iceberg_2 failures total, of which just 4 touch export partition.
  • Even the smoke test .../sanity/export single partition fails, across no catalog / ice catalog / glue catalog and both plain/replicated merge tree → the EXPORT PARTITION operation itself is broken, not a single edge case.

Classification: whole-suite regression, base→PR = 4→842. This is the PR.

🔴 2. S3Export (part) / s3_export_part — PR-caused (same root cause)

  • PR: 66 (release) / 68 (arm) [Fail], spread across export part/{concurrent alter, concurrent other, error handling, pending mutations and patch parts, rbac, table functions}.
  • Base branch: only 5 failures, all in one sub-suite (feature support validation).
  • Same story: base→PR = 5→66. Both features share the Iceberg manifest/data-file writing code this PR reworks.

🟡 3. Stateless tests (arm_binary, parallel) — flaky, not this PR

  • Exactly one test failed: 03518_alter_logical_race [ FAIL ] 182.81 sec — a concurrency/timing stress test (ALTER race) with a documented history of stabilization commits (Let 03518_alter_logical_race extend its budget…, Stabilize 03518_alter_logical_race…).
  • This PR touches nothing in that path (Iceberg/object-storage only). Safe to re-run.

Where the write regression most likely lives

This backport of upstream ClickHouse#90740 re-plumbed the Iceberg write path to support files on secondary/external storages. Every manifest/data-file path is now routed through the new resolveObjectStorageForPath(...) (src/Storages/ObjectStorage/Utils.cpp:962), and a SecondaryStorages is threaded through generateManifestList / writeMetadataFiles / IcebergStorageSink / Iceberg::mutate:

  • IcebergWrites.cpp:929-935 — previous-snapshot manifest list now resolved via resolveObjectStorageForPath instead of the old direct path_resolver.resolve(manifest_list).
  • Mutations.cppwriteMetadataFiles / mutate now require SecondaryStorages &.

For the ordinary single-storage export (no external files — which is exactly what these tests do), that resolver must fall back to the base storage. If it instead throws (e.g. resolveObjectStorageForPathtryResolveObjectStorageForPath reaches Utils.cpp:845 "Unsupported storage scheme" for a serialized relative path that doesn't start with /), or if the manifests are now written with absolute paths the test's pyiceberg readback can't resolve, that breaks every export — matching the wholesale failure.

⚠️ I couldn't fetch the exact DB::Exception text: the regression console only carries the [Fail]/AssertionError summary; the real message lives in the S3 report.html, and web fetch is disabled in my environment. The fastest next step is to open the report and grab the Code: NNN … DB::Exception from the first .../sanity/export single partition failure — that will point straight at the throwing line:

Suggested fixes

  1. Confirm the exact exception from the report (above) — that decides between the two hypotheses.
  2. Make the write path's resolveObjectStorageForPath fall back cleanly to the base storage for serialized relative metadata paths (the single-storage case), rather than throwing / emitting absolute paths. Compare the serialized form fed at IcebergWrites.cpp:932 and Mutations.cpp against what the resolver's relative-path branch (Utils.cpp:577) expects.
  3. Re-check whether any write-path caller passes an empty/half-populated SecondaryStorages where the read path populates it.

I did not push anything. Want me to open a blau/* PR with a proposed fix once we have the exact exception, or do you want to drive it? (I can't build/run ClickHouse here, so any fix I write would be validated by re-running these two regression suites on the PR.)

Evidence: PR run 31795945657 (jobs iceberg_2 94806921483/94805671736, s3_export_part 94806921454/94805672014, stateless 94777903141); base run 31713806632 (jobs 94649901976, 94649902015). I can't build/test locally — this is static + CI-log analysis.

@zvonand

zvonand commented Aug 15, 2026

Copy link
Copy Markdown
Member Author

RelEasy analyze-fails — UNRELATED

run completed at 2026-08-15T05:01:59Z

  • Head SHA: a623cd1625 (feature/antalya-26.6/ClickHouse-ClickHouse-pr-90740)
  • Failed CI checks: 6
  • Tests considered: 381 across 4 CI shard(s)
  • Outcomes: 0 done · 0 partial · 4 unrelated · 0 unresolved
  • Commits added by AI: 0 (—)
  • Anthropic cost: $9.1037
  • Warnings:
    • PR: workflow-level rolled-up report — the per-job statuses cover the same failures; skipping
  • Checks covered by another shard (same failures, investigated once):
    • Regression release iceberg_2: all 326 failure(s) are the same test(s) as Regression aarch64 iceberg_2 — investigated there, no shard of its own.
    • Regression release s3_export_part: 49 of 51 failure(s) duplicate Regression aarch64 s3_export_part; the remaining 2 form this shard.

Per-shard outcomes

⏭️ UNRELATED — Stateless tests (arm_binary, parallel)

1 failed test(s) considered — cost $0.5398
full report

AI narration

I'll start by inspecting the PR diff and the failing test.

Triage

PR scope (gh pr view 2154): "Antalya 26.6: Iceberg: support external paths in tables" — a cherry-pick of ClickHouse#90740. The touched production code is entirely Iceberg / object-storage: src/Storages/ObjectStorage/**, src/IO/S3/URI.*, src/Databases/DataLake/DatabaseDataLake.cpp, plus Iceberg integration tests and minio fixtures. Nothing in MergeTree / ReplicatedMergeTree / ALTER / mutation code paths.

Failure: 03518_alter_logical_race — a ReplicatedMergeTree concurrent ALTER ADD/MODIFY/DROP COLUMN vs INSERT race test. Findings:

  • git diff antalya-26.6..HEAD -- tests/queries/0_stateless/03518_alter_logical_race.{sh,reference} is empty — the test is byte-identical to base; this PR neither added nor touched it. Its last commits (5428fedc12e, a9670c9596b) are already reachable from antalya-26.6.
  • The CI excerpt itself carries the reproducibility verdict: “Failed 0 out of 3 reruns.” — the failure did not reproduce on any rerun on the same binary.
  • The test is explicitly timing-sensitive by construction (soft/hard deadlines, SECONDS-based budgets, progress markers, NO ALTER/INSERT PROGRESS guard). return code: 1 with empty stdout is consistent with the budget/progress guard tripping under load, not with an Iceberg change.
  • I cannot write a one-sentence "this PR broke this test because " — there is no path from Iceberg external-path support to ReplicatedMergeTree ALTER/INSERT scheduling.

Verdict: NOT-THIS-PR.

Verification (no code changed)

Ran the test once against the local server via tests/clickhouse-test 03518_alter_logical_race:

[1 / 1] 03518_alter_logical_race:   [ OK ] 10.70 sec.
1 tests passed. 0 tests skipped.

It passes locally, corroborating the 0/3-rerun reproducibility result.

No build was run (no compiled code touched), no commits were made, and git status --porcelain is empty — the branch is exactly as I found it.

Summary

  • 03518_alter_logical_race[unrelated] — timing-sensitive ReplicatedMergeTree ALTER/INSERT race test, untouched by this PR (identical to antalya-26.6); CI's own reproducibility check failed 0/3 reruns and it passes locally. This PR only changes Iceberg/object-storage code, which the test never exercises.

UNRELATED

Triage

PR scope (gh pr view 2154): "Antalya 26.6: Iceberg: support external paths in tables" — a cherry-pick of ClickHouse#90740. The touched production code is entirely Iceberg / object-storage: src/Storages/ObjectStorage/**, src/IO/S3/URI.*, src/Databases/DataLake/DatabaseDataLake.cpp, plus Iceberg integration tests and minio fixtures. Nothing in MergeTree / ReplicatedMergeTree / ALTER / mutation code paths.

Failure: 03518_alter_logical_race — a ReplicatedMergeTree concurrent ALTER ADD/MODIFY/DROP COLUMN vs INSERT race test. Findings:

  • git diff antalya-26.6..HEAD -- tests/queries/0_stateless/03518_alter_logical_race.{sh,reference} is empty — the test is byte-identical to base; this PR neither added nor touched it. Its last commits (5428fedc12e, a9670c9596b) are already reachable from antalya-26.6.
  • The CI excerpt itself carries the reproducibility verdict: “Failed 0 out of 3 reruns.” — the failure did not reproduce on any rerun on the same binary.
  • The test is explicitly timing-sensitive by construction (soft/hard deadlines, SECONDS-based budgets, progress markers, NO ALTER/INSERT PROGRESS guard). return code: 1 with empty stdout is consistent with the budget/progress guard tripping under load, not with an Iceberg change.
  • I cannot write a one-sentence "this PR broke this test because " — there is no path from Iceberg external-path support to ReplicatedMergeTree ALTER/INSERT scheduling.

Verdict: NOT-THIS-PR.

Verification (no code changed)

Ran the test once against the local server via tests/clickhouse-test 03518_alter_logical_race:

[1 / 1] 03518_alter_logical_race:   [ OK ] 10.70 sec.
1 tests passed. 0 tests skipped.

It passes locally, corroborating the 0/3-rerun reproducibility result.

No build was run (no compiled code touched), no commits were made, and git status --porcelain is empty — the branch is exactly as I found it.

Summary

  • 03518_alter_logical_race[unrelated] — timing-sensitive ReplicatedMergeTree ALTER/INSERT race test, untouched by this PR (identical to antalya-26.6); CI's own reproducibility check failed 0/3 reruns and it passes locally. This PR only changes Iceberg/object-storage code, which the test never exercises.

UNRELATED

⏭️ UNRELATED — Regression aarch64 iceberg_2

326 failed test(s) considered — cost $4.1368
full report

AI narration

…(narration truncated; last 4500 chars)

, no_catalog﹕ icebergS3 table function reads the committed export}·concurrent writes/{INSERT after a scheduled EXPORT does not leak into the snapshot, duplicate EXPORT inside one ALTER commits at most once, multi-statement ALTER commits each partition as its own snapshot}·datatypes/export partition id/accepted/·direct writes/{alternating export and insert, insert after export}·disaster recovery/{EXPORT to a missing destination is rejected synchronously, STOP MOVES holds the export PENDING, START MOVES resumes it}·manifest integrity/{data files have all required column stats, each export advances the snapshot list by one, partition spec references source columns, snapshot summary total-records matches exported row count}·minmax pruning/{minmax pruning on exported data, minmax pruning on range predicate}·partition compatibility/accepted/andpartition compatibility/rejected/·partition spec evolution/{multi-column partition spec is preserved, partition tuple matches partition_id across exports, single spec for multiple partition exports}·sanity/{export all partitions, export empty partition, export multiple partitions in one ALTER, export single partition}·schema compatibility/schema compatibility matrix/(47 example rows) ·schema evolution/{add column between exports, drop column between exports, modify column widen Int32 -> Int64, rename column between exports, source-only schema drift is rejected at EXPORT}·settings/schema mismatch mode/·storage paths/{deep prefix hierarchy round-trips cleanly, default﹕ metadata․json location is bucket-relative (no FS scheme), full path﹕ metadata․json location is an absolute s3﹕∕∕ URI, multiple destinations share a bucket but stay isolated}·system monitoring/·transactions/*·truncate/{export after truncate repopulates destination, truncate after export, truncate after insert}`

B. [unrelated] — 15 tests: unknown client setting export_merge_tree_part_schema_mismatch_mode (exists in neither base nor head).

  • /iceberg/export partition/{glue,ice,no} catalog/replicated merge tree/settings/schema mismatch mode/ignore extra source columns by position drops trailing columns
  • … /settings/schema mismatch mode/ignore extra source columns by position rejects extra destination columns
  • … /settings/schema mismatch mode/ignore extra source columns by position still rejects type mismatch
  • … /settings/schema mismatch mode/strict rejects extra destination columns
  • … /settings/schema mismatch mode/strict rejects extra source columns

C. [unrelated] — 12 tests: server does not reject a repositioned partition-key column (suite expects exit 36); export validation untouched by this PR.

  • /iceberg/export partition/{glue,ice,no} catalog/replicated merge tree/schema compatibility/schema compatibility matrix/present=True, name_same=True, type_same=False, position_same=False, default_same=False, in_partition_key=True, missing_side=None
  • … type_same=False, position_same=False, default_same=True, in_partition_key=True, missing_side=None
  • … type_same=True, position_same=False, default_same=False, in_partition_key=True, missing_side=None
  • … type_same=True, position_same=False, default_same=True, in_partition_key=True, missing_side=None

D. [unrelated] — 11 tests: duplicate-export rejection returns client exit 242 (EXPORT_PARTITION_ALREADY_EXPORTED = 1010), suite allows only 36/238.

  • /iceberg/export partition/{glue,ice,no} catalog/replicated merge tree/concurrent writes/duplicate EXPORT inside one ALTER commits at most once
  • /iceberg/export partition/{glue,ice,no} catalog/replicated merge tree/transactions/duplicate export within TTL is rejected
  • /iceberg/export partition/{glue,ice,no} catalog/replicated merge tree/transactions/TTL expiry permits re-export of the same partition
  • /iceberg/export partition/{glue,ice} catalog/replicated merge tree/multi replica recovery/concurrent exports of the same partition from different replicas are idempotent

E. [unrelated] — 2 tests: iceberg_delete_data_on_drop is a query-level setting but StorageObjectStorage::drop() runs on the global context, so nothing is purged; identical code on antalya-26.6.

  • /iceberg/export partition/ice catalog/replicated merge tree/catalogs/drop with purge allows recreating same table
  • /iceberg/export partition/no catalog/replicated merge tree/catalogs/drop with purge allows recreating same table

UNRELATED

⏭️ UNRELATED — Regression aarch64 s3_export_part

52 failed test(s) considered — cost $1.7075
full report

AI narration

…(narration truncated; last 4500 chars)

nt alter/stress/example=drop constraint[unrelated]` same cascade.

  • /s3/minio/export tests/export part/concurrent alter/stress/example=drop partition[unrelated] same cascade.
  • /s3/minio/export tests/export part/concurrent alter/stress/example=drop table[unrelated] same cascade.
  • /s3/minio/export tests/export part/concurrent alter/stress/example=fetch partition[unrelated] same cascade.
  • /s3/minio/export tests/export part/concurrent alter/stress/example=freeze partition[unrelated] same cascade.
  • /s3/minio/export tests/export part/concurrent alter/stress/example=freeze partition with name[unrelated] same cascade.
  • /s3/minio/export tests/export part/concurrent alter/stress/example=modify column[unrelated] same cascade.
  • /s3/minio/export tests/export part/concurrent alter/stress/example=modify table comment[unrelated] same cascade.
  • /s3/minio/export tests/export part/concurrent alter/stress/example=modify ttl[unrelated] same cascade.
  • /s3/minio/export tests/export part/concurrent alter/stress/example=move partition[unrelated] same cascade.
  • /s3/minio/export tests/export part/concurrent alter/stress/example=move partition to table[unrelated] same cascade.
  • /s3/minio/export tests/export part/concurrent alter/stress/example=rename column[unrelated] same cascade.
  • /s3/minio/export tests/export part/concurrent alter/stress/example=replace partition[unrelated] same cascade.
  • /s3/minio/export tests/export part/concurrent alter/stress/example=unfreeze partition with name[unrelated] same cascade.
  • /s3/minio/export tests/export part/concurrent other/after delete rows/delete_method='ALTER DELETE', delete_condition='i = 1', description='one row'[unrelated] server down cascade.
  • /s3/minio/export tests/export part/concurrent other/after delete rows/delete_method='ALTER DELETE', delete_condition='i IN (1, 3)', description='multiple rows'[unrelated] server down cascade.
  • /s3/minio/export tests/export part/concurrent other/after delete rows/delete_method='ALTER DELETE', delete_condition='p = 1', description='all rows'[unrelated] server down cascade.
  • /s3/minio/export tests/export part/concurrent other/after delete rows/delete_method='DELETE FROM', delete_condition='i = 1', description='one row'[unrelated] server down cascade.
  • /s3/minio/export tests/export part/concurrent other/after delete rows/delete_method='DELETE FROM', delete_condition='i IN (1, 3)', description='multiple rows'[unrelated] server down cascade.
  • /s3/minio/export tests/export part/concurrent other/after delete rows/delete_method='DELETE FROM', delete_condition='p = 1', description='all rows'[unrelated] server down cascade.
  • /s3/minio/export tests/export part/concurrent other/insert parts[unrelated] server down cascade.
  • /s3/minio/export tests/export part/concurrent other/inserts and selects not blocked[unrelated] server down cascade.
  • /s3/minio/export tests/export part/concurrent other/kill export[unrelated] server down cascade.
  • /s3/minio/export tests/export part/concurrent other/merge during export retains parts on disk[unrelated] server down cascade.
  • /s3/minio/export tests/export part/concurrent other/multiple sources same destination[unrelated] server down cascade (verified: CREATE TABLE → Connection refused).
  • /s3/minio/export tests/export part/concurrent other/optimize parts[unrelated] server down cascade.
  • /s3/minio/export tests/export part/concurrent other/select parts[unrelated] server down cascade.
  • /s3/minio/export tests/export part/concurrent other/stress select/select_action=<testflows․_core․test․TestStep object at 0xf073bfd8f7a0>[unrelated] server down cascade.
  • /s3/minio/export tests/export part/concurrent other/stress select/select_action=<testflows․_core․test․TestStep object at 0xf073bfeae000>[unrelated] server down cascade.
  • /s3/minio/export tests/export part/pending mutations and patch parts/pending mutations allow[unrelated] server down cascade.
  • /s3/minio/export tests/export part/pending mutations and patch parts/pending patch parts allow[unrelated] server down cascade.
  • /s3/minio/export tests/export part/rbac/alter insert privilege[unrelated] server down cascade.
  • /s3/minio/export tests/export part/table functions[unrelated] server down cascade.

No commits made; nothing to revert.

UNRELATED

⏭️ UNRELATED — Regression release s3_export_part

2 failed test(s) considered — cost $2.7196
full report

AI narration

…(narration truncated; last 4500 chars)

nnection refused that is fully explained by the logs, and the scenario itself never ran a single export or select. No code was changed; no commits added; working tree clean (git status --porcelain` empty).

Per-test verdict

  • /s3/minio/export tests/export part/concurrent other/stress select/select_action=<testflows․_core․test․TestStep object at 0x76c07e282480>[unrelated] — failed in 147 ms at CREATE TABLE with Connection refused (localhost:9000); the server was left dead by the earlier during minio interruption scenario (S3 disk startup failed: Not found address of host: minio1 (DNS_ERROR)), and this PR only changes Iceberg-gated code paths that this plain-S3/Parquet export test never touches.
  • /s3/minio/export tests/export part/concurrent other/stress select/select_action=<testflows․_core․test․TestStep object at 0x76c07e2ca180>[unrelated] — identical cascade, same 136 ms setup failure; sibling example of the same outline, same root cause.

UNRELATED

Investigation summary

Diff triage. The PR (Antalya 26.6: Iceberg: support external paths in tables, backport of ClickHouse#90740) touches only Iceberg/object-storage path resolution: Iceberg/*, IObjectIterator, ReadBufferIterator, StorageObjectStorageSource, StorageObjectStorageStableTaskDistributor, ObjectStorage/Utils.{cpp,h} (purely additive — no - lines), plus a defaulted extra arg on S3::URI (enable_url_encoding = true, matching Poco's own default) and a class rename in StorageURL.cpp. Every behavioural change is gated behind dynamic_pointer_cast<IcebergDataObjectInfo> (getResolvedStorageFromObjectInfo / getMetadataPathFromObjectInfo return the base storage / nullopt for plain objects), so non-Iceberg S3 reads and writes are untouched.

The failing scenario (s3/tests/export_part/concurrent_other.py::stress_select) exports MergeTree parts into a plain S3(..., format='Parquet', partition_strategy='hive') destination — no Iceberg anywhere.

Root cause from CI logs (fetched fails.log.txt / nice-new-fails.log.txt from the artifact bucket): the shard reports 18 failed scenarios, of which only these 2 were "new". Both stress select examples failed after 136 ms / 147 ms, in their very first Given step:

[clickhouse1] > ... ENGINE = MergeTree ORDER BY tuple() PARTITION BY p; ...
[clickhouse1] Code: 210. DB::NetException: Connection refused (localhost:9000). (NETWORK_ERROR)

The server was already dead. It died earlier, in concurrent alter/during minio interruption/example=replace partition: that scenario kills the MinIO container, then restarts ClickHouse to apply a config change, and the server refused to come back up:

<Error> void DB::IDisk::startup(bool): Code: 499. ... Code: 198. DB::NetException: Not found address of host: minio1. (DNS_ERROR)
<Error> Application: Caught exception while loading metadata: ... Not found address of host: minio1. (DNS_ERROR)

From that point every remaining scenario in the module (concurrent alter/stress/*, all of concurrent other/* including both stress select examples, after delete rows/*, rbac, …) failed instantly with the same connection-refused cascade. This is a test-harness/environment failure (MinIO container killed by a sibling scenario, server unable to start its S3 disk), not a product regression, and it is entirely independent of the PR's diff.

No reproduction attempt was made against a live cluster: the failure is a setup-time Connection refused that is fully explained by the logs, and the scenario itself never ran a single export or select. No code was changed; no commits added; working tree clean (git status --porcelain empty).

Per-test verdict

  • /s3/minio/export tests/export part/concurrent other/stress select/select_action=<testflows․_core․test․TestStep object at 0x76c07e282480>[unrelated] — failed in 147 ms at CREATE TABLE with Connection refused (localhost:9000); the server was left dead by the earlier during minio interruption scenario (S3 disk startup failed: Not found address of host: minio1 (DNS_ERROR)), and this PR only changes Iceberg-gated code paths that this plain-S3/Parquet export test never touches.
  • /s3/minio/export tests/export part/concurrent other/stress select/select_action=<testflows․_core․test․TestStep object at 0x76c07e2ca180>[unrelated] — identical cascade, same 136 ms setup failure; sibling example of the same outline, same root cause.

UNRELATED


🤖 Posted automatically by releasy analyze-fails. Re-run the command to refresh.

@zvonand

zvonand commented Aug 15, 2026

Copy link
Copy Markdown
Member Author

CI triage — none of the failures are caused by this PR

There are two contradicting triage comments above. I re-derived the answer from the raw testflows artifacts rather than from either summary. Result: all failing checks are pre-existing on antalya-26.6; zero failures are attributable to this PR.

Why the export suites suddenly went red

Both export suites are gated behind a build-feature probe. On every recent base-branch MasterCI run the probe reported:

Export merge tree partition is not supported by this build; the feature is not present in this version.

The gate is /iceberg/export partition/feature support validation, which checks for the export_merge_tree_partition_force_export setting. At base SHA a894ddeb that identifier exists only in SettingsChangesHistory.cpp, not in Settings.cpp — so the suite was skipped and zero export-partition scenarios ever executed on base.

That feature reached antalya-26.6 with #2146 (Antalya 26.6: Partition export + cluster functions), merged 2026-08-14T08:39Zafter the last base MasterCI run (2026-08-13). This PR merged origin/antalya-26.6 in ff48b66 and therefore picked up #2146, so its CI ran the suite for the first time.

Control PRs on the same merge base

Comparing sorted leaf-Fail sets from fails.log.txt:

PR Subject (unrelated to Iceberg path resolution) iceberg_2 fails md5 of sorted set
2154 this PR 422 684030a9…
2210 commit info in partition exports table 422 684030a9…
2209 export task timeout 24h 422 684030a9…
2156 cache vended REST-catalog credentials 422 684030a9…
2208 use_client_time_zone backport 422 684030a9…
2201 / 2200 / 2193 (pre-#2146) 7 suite gated off

Byte-identical failure sets on both x86_64 and aarch64. Four PRs containing none of this backport's code produce exactly the same 422 failures, including every sanity/export single partition case.

s3_export_part

Same picture: 2154 → 66, 2210 → 66, 2209/2208 → 68. Pre-#2146 PRs (2200, 2201) → 5, and those 5 are only /s3, /s3/minio, /s3/minio/export tests, /s3/minio/export tests/export part and its feature support validation gate.

Two entries initially looked unique to this PR, but the test name embeds a Python object address (<testflows._core.test.TestStep object at 0x76c07e282480>) that changes every run. After normalizing it, only-in-2154 = 0 against every control; the controls carry extra failures this PR does not have, because the MinIO-interruption cascade lands on a different scenario each run (concurrent alter/during export vs after export).

Stateless tests (arm_binary, parallel)

03518_alter_logical_race — failed 0 out of 3 reruns. Flaky, unrelated.

Correction to the earlier "PR-caused" analysis

That comment compared this PR against MasterCI run 31713806632 and read base's "4 export-partition failures" as 4 executed scenarios. Those 4 lines are the gate plus its parent suite rows, so base→PR = 4→842 compares suite skipped against suite ran and carries no information. The same applies to 5→66 on s3_export_part. Its claim that base fails 8 iceberg engine read scenarios is also a misread — base's only other failure is the iceberg iterator race condition gate (object_storage_cluster_join_mode missing from that build) plus parents. The proposed root cause in resolveObjectStorageForPath is directly refuted: PRs that contain none of that code show identical failures.

Caveats

  • This is artifact-level attribution — it shows the PR does not cause these failures. It is not a correctness review of the backport itself.
  • On the positive side, /iceberg/iceberg engine (the read path this PR targets) has no failures here, while base a894ddeb failed its iceberg iterator race condition gate. All 422 failures sit under /iceberg/export partition.
  • Separately worth flagging: Antalya 26.6: Partition export + cluster functions #2146 landed on antalya-26.6 with 422 failing iceberg_2 scenarios and ~62 failing s3_export_part scenarios. Until that is addressed, both suites are non-signal on every PR against this branch — they will not catch a genuine regression either. There has been no base MasterCI run since Antalya 26.6: Partition export + cluster functions #2146 merged, so this is not yet recorded as the new baseline.

Related: #2146

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-resolved Port conflict auto-resolved by Claude antalya-26.6 releasy Created/managed by RelEasy

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants