[split 13/22] db: DBMetadata engine detection robustness, BaseDbWriter/Hikari connection lifecycle - #1368
Closed
minguyen9988 wants to merge 6 commits into
Closed
Conversation
…ting fixes Builds on the config PR (merge after it). - QueryFormatter.getInsertQueryUsingInputFunction: new overload accepting the destination's known columns so destination-only columns (e.g. added via ALTER TABLE ADD COLUMN before the connector saw the DDL) are OMITTED from the insert - ClickHouse then fills their DEFAULT - instead of force-binding NULL, which poisons the whole batch on non-nullable columns and replays it forever (observed as a 165/200 data-loss failure on the combined PR). - ClickHouseAlterTable / ClickHouseAutoCreateTable / ClickHouseCreateDatabase: identifier quoting per dotted component; ClickHouseDbConstants gains the shared column-name constants. - DbKafkaOffsetWriter aligned with the quoting rules. - QueryFormatterTest + ClickHouseAlterTableTest expanded to pin all of the above. Part of the split of #1353 into independently mergeable sub-PRs (each <= 10 files), so the 2.10.0 branch can absorb the fixes incrementally.
…n regression suite - The built-in ANALYZE PARTITION ignore pattern gains (?i) (its ADD/DROP PARTITION siblings already had it), and the built-in ignore branch now records lastIgnoredDDL exactly like the user-configured IGNORE_DDL_REGEX branch - previously a built-in-ignored statement was invisible to operators. - MySqlDDLParserListenerImpl: DATETIME(p) precision translation, generated-column and charset clause handling, extractTableName helper. - DDLTranslationRegressionTest (new) + DropTruncateDetectionTest pin MySQL -> ClickHouse DDL translation end-to-end; MySqlDDLParserListenerImplTest expanded. - DataTypeConverter DateTime64 precision fix rides along because the translation tests assert through it. Part of the split of #1353 into independently mergeable sub-PRs (each <= 10 files), so the 2.10.0 branch can absorb the fixes incrementally.
…plication freeze manager, error classifier (new classes, not yet wired) Adds four new core classes with their unit tests. NOT yet referenced by any existing code path - wiring happens in the executor PR - so this PR is behavior-neutral by construction. - DDLSchemaChangeWaiter: after a DDL, polls system.columns (bounded by ddl.schema.change.timeout.ms) until the destination schema reflects the change, so DML following a DDL never binds against a stale column set. Exempts the replication-history table (its audit schema never mirrors source columns; gating on it starved the shared system pool on the combined PR). - SourceSchemaIntegrityValidator + SourceSchemaColumns: compares source-event columns against the destination insert cache; genuinely missing columns block the batch (data-loss protection), while MySQL generated columns (MATERIALIZED on the destination, excluded from the insert cache by design) are filtered out - counting them as missing caused an 8,208-rebuild livelock on the combined PR. - TableReplicationFreezeManager: freezes replication for a table on unrecoverable schema mismatch instead of silently dropping records. - ClickHouseErrorClassifier: classifies ClickHouse exceptions into retryable vs fatal so the batch loop can stop retrying permanently-failing batches. Merge order: after the config PR (uses ddl.schema.change.* keys). Part of the split of #1353 into independently mergeable sub-PRs (each <= 10 files), so the 2.10.0 branch can absorb the fixes incrementally.
…ATE guard on CDC path, version sentinel guard Merge after the db-ops PR (uses the QueryFormatter overload) and the config PR (reads disable.drop.truncate). - PreparedStatementFieldMapper / PreparedStatementExecutor: destination-only columns are omitted from binds (paired with the QueryFormatter overload); the CDC data path now honors disable.drop.truncate before executing TRUNCATE - previously this path applied every TRUNCATE unconditionally, so the guard silently failed to protect it. - GroupInsertQueryWithBatchRecords: passes the destination column set through (new overload; the old signature is kept); dead failedRecords list removed (was never read). - VersionSentinelGuardTest: pins that a missing version column can never silently bind the 0 sentinel (_version=0 rows lose every ReplacingMergeTree merge - silent row loss). Part of the split of #1353 into independently mergeable sub-PRs (each <= 10 files), so the 2.10.0 branch can absorb the fixes incrementally.
…che invalidation generations, schema-cache TTL Wires the new core classes into the batch path. The heart of the race-condition fixes. Merge after config, db-ops, core-classes and batch PRs. - Duplicate-insert fix: an already-flushed batch waiting on offset commit was re-INSERTED on every retry pass (one record observed 1,000 times on the combined PR). A flushed batch now only re-checks committability - never re-inserts. - ClickHouseBatchExecutor.isPaused becomes volatile: it is the DDL-vs-DML barrier, written by the Debezium thread and read in the batch threads' spin loop with no other happens-before edge. Without volatile the read can be hoisted and a batch thread can apply DML against a schema mid-DDL - silent corruption, not a stall. - Cache invalidation by generation: the previous remove-on-read signal let the FIRST worker thread consume the invalidation, leaving every other thread on a stale DbWriter (stale column list) forever. Per-table monotonic generation counters let every thread invalidate independently; plus a schema-cache TTL (default 1h) so a missed DDL self-heals. CacheInvalidationManagerTest / CacheInvalidationManagerTtlTest. - DebeziumOffsetManagement.acknowledgeRecord for the single-record path; beginFlush no longer collides with an in-flight flush (the original OffsetStorageWriter "already flushing" crash from the production log). - Integrity gate + freeze manager + error classifier wired into ClickHouseBatchRunnable / ClickHouseBatchWriter; DbWriter carries the cache generation. Part of the split of #1353 into independently mergeable sub-PRs (each <= 10 files), so the 2.10.0 branch can absorb the fixes incrementally.
…ction lifecycle - DBMetadata: engine detection (getTableEngine / RMT version-column parsing) hardened against unexpected system.tables output; result-set and statement lifecycles closed via try-with-resources. DBMetadataTest rewritten accordingly. - BaseDbWriter: connection-creation failures logged with target-URL context and rethrown (previously half-swallowed); shared SYSTEM_DB constant. - HikariDbSource: pool lookup for a database that was never initialized returns a clear error instead of NPE (HikariDbSourceTest). - ErrorLogger format fix. Part of the split of #1353 into independently mergeable sub-PRs (each <= 10 files), so the 2.10.0 branch can absorb the fixes incrementally.
minguyen9988
force-pushed
the
omniwatcher/split-13-db-metadata
branch
from
August 6, 2026 00:13
3e20def to
fce4305
Compare
minguyen9988
changed the base branch from
2.10.0
to
omniwatcher/split-20-executor-cache
August 6, 2026 00:13
Base automatically changed from
omniwatcher/split-20-executor-cache
to
omniwatcher/split-19-batch-statement
August 18, 2026 14:34
Base automatically changed from
omniwatcher/split-19-batch-statement
to
omniwatcher/split-18-core-new-classes
August 18, 2026 14:34
Base automatically changed from
omniwatcher/split-18-core-new-classes
to
omniwatcher/split-17-ddl-parser
August 18, 2026 14:34
Base automatically changed from
omniwatcher/split-17-ddl-parser
to
omniwatcher/split-16-db-ops
August 18, 2026 14:34
Base automatically changed from
omniwatcher/split-16-db-ops
to
omniwatcher/split-15-sink-config
August 18, 2026 14:35
Collaborator
Author
|
Closing this split PR at the request of the Jump DBA team. The series is being retired in favour of a consolidated approach; the branch is left in place for reference. Thanks to the maintainers for the reviews on the merged parts of the series. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of the split of #1353 into independently mergeable sub-PRs (each <= 10 files), so the 2.10.0 branch can absorb the fixes incrementally.
Split out of #1353, which this series replaces. Each sub-PR is <= 10 files; the union of all 22 reproduces the #1353 tree exactly (verified by tree SHA).