[split 14/22] tests: end-to-end DDL data-integrity IT harness (Docker-gated, test-only) - #1369
Closed
minguyen9988 wants to merge 8 commits into
Closed
Conversation
…ting fixes Builds on the config PR (merge after it). - QueryFormatter.getInsertQueryUsingInputFunction: new overload accepting the destination's known columns so destination-only columns (e.g. added via ALTER TABLE ADD COLUMN before the connector saw the DDL) are OMITTED from the insert - ClickHouse then fills their DEFAULT - instead of force-binding NULL, which poisons the whole batch on non-nullable columns and replays it forever (observed as a 165/200 data-loss failure on the combined PR). - ClickHouseAlterTable / ClickHouseAutoCreateTable / ClickHouseCreateDatabase: identifier quoting per dotted component; ClickHouseDbConstants gains the shared column-name constants. - DbKafkaOffsetWriter aligned with the quoting rules. - QueryFormatterTest + ClickHouseAlterTableTest expanded to pin all of the above. Part of the split of #1353 into independently mergeable sub-PRs (each <= 10 files), so the 2.10.0 branch can absorb the fixes incrementally.
…n regression suite - The built-in ANALYZE PARTITION ignore pattern gains (?i) (its ADD/DROP PARTITION siblings already had it), and the built-in ignore branch now records lastIgnoredDDL exactly like the user-configured IGNORE_DDL_REGEX branch - previously a built-in-ignored statement was invisible to operators. - MySqlDDLParserListenerImpl: DATETIME(p) precision translation, generated-column and charset clause handling, extractTableName helper. - DDLTranslationRegressionTest (new) + DropTruncateDetectionTest pin MySQL -> ClickHouse DDL translation end-to-end; MySqlDDLParserListenerImplTest expanded. - DataTypeConverter DateTime64 precision fix rides along because the translation tests assert through it. Part of the split of #1353 into independently mergeable sub-PRs (each <= 10 files), so the 2.10.0 branch can absorb the fixes incrementally.
…plication freeze manager, error classifier (new classes, not yet wired) Adds four new core classes with their unit tests. NOT yet referenced by any existing code path - wiring happens in the executor PR - so this PR is behavior-neutral by construction. - DDLSchemaChangeWaiter: after a DDL, polls system.columns (bounded by ddl.schema.change.timeout.ms) until the destination schema reflects the change, so DML following a DDL never binds against a stale column set. Exempts the replication-history table (its audit schema never mirrors source columns; gating on it starved the shared system pool on the combined PR). - SourceSchemaIntegrityValidator + SourceSchemaColumns: compares source-event columns against the destination insert cache; genuinely missing columns block the batch (data-loss protection), while MySQL generated columns (MATERIALIZED on the destination, excluded from the insert cache by design) are filtered out - counting them as missing caused an 8,208-rebuild livelock on the combined PR. - TableReplicationFreezeManager: freezes replication for a table on unrecoverable schema mismatch instead of silently dropping records. - ClickHouseErrorClassifier: classifies ClickHouse exceptions into retryable vs fatal so the batch loop can stop retrying permanently-failing batches. Merge order: after the config PR (uses ddl.schema.change.* keys). Part of the split of #1353 into independently mergeable sub-PRs (each <= 10 files), so the 2.10.0 branch can absorb the fixes incrementally.
…ATE guard on CDC path, version sentinel guard Merge after the db-ops PR (uses the QueryFormatter overload) and the config PR (reads disable.drop.truncate). - PreparedStatementFieldMapper / PreparedStatementExecutor: destination-only columns are omitted from binds (paired with the QueryFormatter overload); the CDC data path now honors disable.drop.truncate before executing TRUNCATE - previously this path applied every TRUNCATE unconditionally, so the guard silently failed to protect it. - GroupInsertQueryWithBatchRecords: passes the destination column set through (new overload; the old signature is kept); dead failedRecords list removed (was never read). - VersionSentinelGuardTest: pins that a missing version column can never silently bind the 0 sentinel (_version=0 rows lose every ReplacingMergeTree merge - silent row loss). Part of the split of #1353 into independently mergeable sub-PRs (each <= 10 files), so the 2.10.0 branch can absorb the fixes incrementally.
…che invalidation generations, schema-cache TTL Wires the new core classes into the batch path. The heart of the race-condition fixes. Merge after config, db-ops, core-classes and batch PRs. - Duplicate-insert fix: an already-flushed batch waiting on offset commit was re-INSERTED on every retry pass (one record observed 1,000 times on the combined PR). A flushed batch now only re-checks committability - never re-inserts. - ClickHouseBatchExecutor.isPaused becomes volatile: it is the DDL-vs-DML barrier, written by the Debezium thread and read in the batch threads' spin loop with no other happens-before edge. Without volatile the read can be hoisted and a batch thread can apply DML against a schema mid-DDL - silent corruption, not a stall. - Cache invalidation by generation: the previous remove-on-read signal let the FIRST worker thread consume the invalidation, leaving every other thread on a stale DbWriter (stale column list) forever. Per-table monotonic generation counters let every thread invalidate independently; plus a schema-cache TTL (default 1h) so a missed DDL self-heals. CacheInvalidationManagerTest / CacheInvalidationManagerTtlTest. - DebeziumOffsetManagement.acknowledgeRecord for the single-record path; beginFlush no longer collides with an in-flight flush (the original OffsetStorageWriter "already flushing" crash from the production log). - Integrity gate + freeze manager + error classifier wired into ClickHouseBatchRunnable / ClickHouseBatchWriter; DbWriter carries the cache generation. Part of the split of #1353 into independently mergeable sub-PRs (each <= 10 files), so the 2.10.0 branch can absorb the fixes incrementally.
…ibility (test-only) Test-only. Merge after the executor PR. Multi-threaded race harnesses: concurrent DDL + batch threads against the schema-change waiter (DDLSchemaChangeWaiterRaceConditionTest, DDLFreezeRaceConditionTest), concurrent offset acknowledgement vs flush (OffsetCommitConcurrencyTest), executor pause visibility under the JMM (ExecutorPauseVisibilityTest), and lazy system-connection construction (LazySystemConnectionTest). Split from the executor PR only to respect the 10-file-per-PR limit. Part of the split of #1353 into independently mergeable sub-PRs (each <= 10 files), so the 2.10.0 branch can absorb the fixes incrementally.
…tion, restart-loop fixes, sequence race Final wiring PR for the lightweight connector; merge last. - Single-threaded NPE: in single.threaded mode no batch executor exists, but the DDL branch called executor.pause() unconditionally - NPE on the very first DDL of every run (upstream swallowed it in a log-only catch). Null-safe pause/resume helpers; in single-threaded mode DML is synchronous on the DDL thread, so the no-op is semantically correct. SingleThreadedDdlPauseTest. - Snapshot DDL exemption: the (now actually working) disable.drop.truncate guard suppressed the DROP half of Debezium's snapshot DROP+CREATE replay, so every CREATE hit TABLE_ALREADY_EXISTS in a retry freeze. Snapshot-phase DDL is exempt - its delivery is already governed by enable.snapshot.ddl; streaming DROP/TRUNCATE stays blocked. DropTruncateSnapshotExemptionTest pins all four quadrants. - Restart handling: engine rebuild moved off the callback thread (inline rebuild nested a stack frame per retry); stop() uses shutdownNow(); upstream retry budget restored (11 attempts). DebeziumEngineRestartTest. - Sequence race: out-of-order batches could re-arm the version-counter reset so two rows shared one _version (measured 199/200 duplicates) - the anchor no longer moves backward. SequenceNumberRaceTest. - IgnoreDDLRegexLoader case handling + IgnoreDDLRegexLoaderTest; Version280CompatibilityTest pins the 2.8.0 _version formula byte-for-byte. Part of the split of #1353 into independently mergeable sub-PRs (each <= 10 files), so the 2.10.0 branch can absorb the fixes incrementally.
…nly) Test-only; requires Docker (runs as IT in CI, skipped where unavailable). EndToEndDDLDataIntegrityIT drives a MySQL -> connector -> ClickHouse pipeline through an interleaved DDL+DML script (e2e_ddl_integrity.sql) - ADD/DROP/MODIFY COLUMN, RENAME, TRUNCATE-guard - asserting row-count and value-level equality after each phase. README documents the harness. Part of the split of #1353 into independently mergeable sub-PRs (each <= 10 files), so the 2.10.0 branch can absorb the fixes incrementally.
minguyen9988
force-pushed
the
omniwatcher/split-14-e2e-ddl-it
branch
from
August 5, 2026 21:01
2470d90 to
265e26a
Compare
minguyen9988
changed the base branch from
2.10.0
to
omniwatcher/split-22-lw-engine
August 5, 2026 21:02
Collaborator
Author
|
Restacked onto the split-15..22 chain (base now |
Base automatically changed from
omniwatcher/split-22-lw-engine
to
omniwatcher/split-21-core-race-tests
August 18, 2026 14:33
Base automatically changed from
omniwatcher/split-21-core-race-tests
to
omniwatcher/split-20-executor-cache
August 18, 2026 14:33
Base automatically changed from
omniwatcher/split-20-executor-cache
to
omniwatcher/split-19-batch-statement
August 18, 2026 14:34
Base automatically changed from
omniwatcher/split-19-batch-statement
to
omniwatcher/split-18-core-new-classes
August 18, 2026 14:34
Base automatically changed from
omniwatcher/split-18-core-new-classes
to
omniwatcher/split-17-ddl-parser
August 18, 2026 14:34
Base automatically changed from
omniwatcher/split-17-ddl-parser
to
omniwatcher/split-16-db-ops
August 18, 2026 14:34
Base automatically changed from
omniwatcher/split-16-db-ops
to
omniwatcher/split-15-sink-config
August 18, 2026 14:35
Collaborator
Author
|
Closing this split PR at the request of the Jump DBA team. The series is being retired in favour of a consolidated approach; the branch is left in place for reference. Thanks to the maintainers for the reviews on the merged parts of the series. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Test-only; requires Docker (runs as IT in CI, skipped where unavailable).
EndToEndDDLDataIntegrityIT drives a MySQL -> connector -> ClickHouse pipeline through an interleaved DDL+DML script (e2e_ddl_integrity.sql) - ADD/DROP/MODIFY COLUMN, RENAME, TRUNCATE-guard - asserting row-count and value-level equality after each phase. README documents the harness.
Part of the split of #1353 into independently mergeable sub-PRs (each <= 10 files), so the 2.10.0 branch can absorb the fixes incrementally.
Split out of #1353, which this series replaces. Each sub-PR is <= 10 files; the union of all 22 reproduces the #1353 tree exactly (verified by tree SHA).