[server] add lake-aware partition drop when auto-partition retention is enabled

[zuston]

Purpose

Auto-partition retention currently drops expired partitions immediately. For lake-enabled tables, lake tiering may still be catching up, which can cause the latest records to be deleted before they are committed to the lake.

Brief change log

This PR introduces an opt-in table.auto-partition.drop.ensure-tiered option. When enabled:

1. The coordinator freezes writes on all bucket leaders and records this frozen status for partition registration in ZooKeeper.
2. It waits until the lake snapshot reaches the recorded log end offset for every bucket.
3. The expired partition is dropped only after all frozen data has been tiered successfully.

And for the following corner cases, this PR also has well design

Coordinator failover: All in-memory partition-drop states are lost. Any expired partition that was being frozen, waiting for lake tiering, or ready to be dropped will be rediscovered by AutoPartitionManager and restarted from FREEZING. Because freezing is monotonic and idempotent, the new Coordinator can safely recapture the frozen offsets, repeat the lake-tiering check, and eventually drop the partition.

Bucket leader failover: Before a follower is promoted to leader, it reads the current PartitionRegistration from ZooKeeper. If the partition is marked as frozen, the new leader restores the write fence before accepting writes. Therefore, a leader change cannot reopen the partition for writes, and the drop workflow can continue safely.

Tests

API and Format

Documentation

[luoyuxia]

Thanks for the pr. I promise I'll have a look

[zuston]

Thanks for the pr. I promise I'll have a look

thanks for your attention! @luoyuxia

One concern is that the drop-state transition is currently driven by the external AutoPartitionManager. If auto-retention is disabled for a table after one of its partitions has entered the FREEZING state, that partition may remain stuck indefinitely.

I do not yet have a good way to handle this without adding a built-in driver—possibly a scheduled executor or thread pool—inside LakeAwarePartitionDropManager. As an initial step, we could dynamically disallow retention changes whenever lake-aware retention is enabled.