[server] batch fetch latest bucket snapshots to reduce latency

[zuston]

Purpose

For primary-key tables, tiering split generation fetches the latest KV snapshot for every bucket in each active partition. The previous implementation performed two sequential ZooKeeper reads per bucket: listing all snapshot IDs and then reading the latest snapshot data.
With 13 partitions and 800 buckets, this resulted in up to 20,800 sequential ZooKeeper reads, causing split generation to take around 450–490 seconds. This delay also extended the actual duration of each tiering round.

Brief change log

Change snapshot loading to two pipelined phases:

1. Fetch snapshot children for all buckets concurrently.
2. Determine the latest snapshot ID for each bucket in memory.
3. Fetch the corresponding snapshot data concurrently.

The implementation reuses the existing background read utilities and respects the configured maximum inflight requests. Missing or concurrently deleted snapshot nodes are handled gracefully, while the existing result semantics and upper-layer interfaces remain unchanged.

Tests

API and Format

Documentation

[zuston]

@luoyuxia @loserwang1024 could you help review this?

[Copilot]

Pull request overview

This PR optimizes Fluss server ZooKeeper snapshot lookup during tiering split generation by batching “latest bucket snapshot” reads into two pipelined phases (list children → fetch latest snapshot data), reducing end-to-end latency while keeping the existing result semantics.

Changes:

• Refactored bucket snapshot loading to batch getChildren for all buckets, compute latest snapshot IDs in-memory, then batch getData for only the latest snapshot nodes.
• Made getDataInBackground(...) visible for testing to enable race-condition simulation.
• Added/extended ZooKeeperClient tests covering batch snapshot behavior, inflight-limit scenarios, and deletion races between the two batch phases.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
fluss-server/src/main/java/org/apache/fluss/server/zk/ZooKeeperClient.java	Batch/pipeline latest-bucket-snapshot reads using existing background request utilities.
fluss-server/src/test/java/org/apache/fluss/server/zk/ZooKeeperClientTest.java	Adds tests for batched snapshot fetching, inflight throttling, and deletion race handling.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.