fix(cdc): apply DDL at its position in the stream, and make the pause flag visible

[minguyen9988]

Problem

Two defects let rows captured against a pre-DDL schema reach ClickHouse after the schema change.

1. Ordering inversion in handleBatch — the primary defect

Rows are accumulated into a local list and handed to the consumers only after the loop over the Debezium batch completes, while a DDL inside that same loop is applied to ClickHouse synchronously:

List<ClickHouseStruct> batch = new ArrayList<>();
for (int i = 0; i < list.size(); i++) {
    ...
    ClickHouseStruct chStruct = processEveryChangeRecord(...);  // <- DDL executes HERE
    if (chStruct != null) {
        batch.add(chStruct);
    }
}
...
if (batch.size() > 0) {
    appendToRecords(batch, config);                             // <- rows enqueued HERE
}

For a batch of [row1, row2, ALTER, row3] the ALTER lands first, and rows 1–2 — captured against the pre-ALTER schema — are written after it.

This is guaranteed by control flow, not by thread timing. It reproduces on every batch carrying a DDL after at least one row; no race is required.

The consequence is visible in the logs. The INSERT column list is built from the ClickHouse column map, so a row lacking the new field trips DataException in PreparedStatementFieldMapper and is written with NULL, emitting these two lines per record:

ERROR: ClickHouse column <c> not present in source
ERROR: Setting column <c> to NULL might fail for non-nullable columns

Around 160k such lines were observed across a single ALTER window under load. Against a nullable column the write is benign; against a non-nullable one it is exactly the failure the second line warns about.

2. Non-volatile pause flag

ClickHouseBatchExecutor.isPaused is written by the Debezium event thread in pause()/resume() and read by every pool thread in beforeExecute, with no happens-before edge between them. A pool thread was not guaranteed to observe either transition — it could miss the pause and begin a batch during a DDL, or spin past the resume and stall.

Fix

1. Pending rows are handed over before the DDL is applied, so the schema change lands at its true position in the stream.
2. isPaused is declared volatile.

What this does not claim to fix

pause() still only gates task start via beforeExecute — it never interrupts an in-flight processBatch and never drains the queue. Fix 1 is what corrects the ordering; fix 2 makes the existing gate behave as written rather than by luck. A real per-table barrier is a larger change and is deliberately not attempted here, so that this PR stays reviewable and its two changes stay independently assessable.

Implementation note

isDDLRecord mirrors the DDL branch's own test: a value struct with a DDL field holding a non-empty statement. The field name is matched case-insensitively but read back by the name that matched — Struct.get is case-sensitive and throws on a different spelling. A hardcoded lowercase read silently classified an uppercase DDL field as a row event; the tests caught this during development, and both spellings are now asserted.

The check is exception-safe: an unreadable record is treated as non-DDL, preserving the previous behaviour rather than breaking the batch loop.

Tests

Four cases in DebeziumChangeEventCaptureTest:

Test	Covers
shouldRecogniseDDLRecord	both DDL and ddl field spellings
shouldNotTreatRowChangeAsDDL	a row-change event
shouldNotTreatEmptyDDLAsDDL	empty, absent and null statements

Both directions matter: a false negative reintroduces the inversion, a false positive flushes on every row and defeats batching.

Full class: Tests run: 16, Failures: 0, Errors: 0, Skipped: 0

Verification status — please read

The predicate driving the fix is unit-tested and the class passes. What is not yet verified here is the end-to-end effect — a live MySQL → connector → ClickHouse run issuing DDL under concurrent write load, asserting the "column not present in source" volume drops and no row loses the new column's value.

The integration suite is testcontainers-based and the environment used to prepare this change cannot pull the required images, so that run has not happened. This is filed as a draft for that reason. I will attach the end-to-end result — or a correction — rather than leave the gap unstated.

How this was found

End-to-end verification of the 2.10.0 rollup against a MySQL → connector → ClickHouse test environment: 14 concurrent threads issuing DML while DDL ran against the same tables, comparing values rather than row counts. Reproduced identically on 2.8.0 and 2.10.0 (165,861 and 159,583 error lines respectively).