RANGER-5744: Harden plugins-docker-build against Ozone KDC startup race

[ramackri]

Summary

Fixes intermittent plugins-docker-build CI failures where ozone-datanode exits during Kerberos login before the Ranger KDC is fully ready in the Docker Compose smoke stack.

Example failure: https://github.com/apache/ranger/actions/runs/31185432016/job/93444713551

Root cause: Ozone datanode/scm started on ranger-kdc: service_started while Java Kerberos clients probe KDC over UDP first; the existing KDC healthcheck only validated TCP port 88. Re-running the job usually passes once the KDC is warm.

Changes

• KDC healthcheck (Dockerfile.ranger-kdc): replace TCP nc probe with kinit using the datanode keytab so healthy means Kerberos actually works
• Compose ordering (docker-compose.ranger-ozone.yml): Ozone datanode/scm/om wait for ranger-kdc: service_healthy (same pattern as ranger-zk)
• krb5.conf: set udp_preference_limit = 0 to avoid UDP PortUnreachableException in Docker CI when TCP is ready first (see below)
• Startup script (ozone-service-start.sh): wait for keytab file and KDC reachability before starting Ozone SCM/datanode
• Restart policy: restart: on-failure:3 on ozone-datanode and ozone-scm for one-shot race recovery
• CI workflow (.github/workflows/ci.yml): explicit wait for ranger-kdc healthy before the container status check

udp_preference_limit = 0 in krb5.conf

Defined once in dev-support/ranger-docker/scripts/kdc/krb5.conf and shared across the Docker CI stack (mounted into Ranger Admin, ZK, Hadoop, Hive, HBase, Kafka, Knox, Ozone, OpenSearch, Solr, KMS, PDP, UserSync, TagSync; also baked into the KDC image). It does not affect production Ranger installs outside dev-support/ranger-docker.

What it does

udp_preference_limit controls how Kerberos clients reach the KDC. By default, clients (Java kinit, Hadoop, Ozone, etc.) try the KDC over UDP port 88 first, then fall back to TCP if UDP fails.

Setting:

udp_preference_limit = 0

means the client never uses UDP for KDC ticket requests (AS-REQ / TGS-REQ). All KDC communication uses TCP port 88 only.

Per MIT Kerberos: “If this flag is set to 0, UDP will not be used to send messages to the KDC.”

flowchart LR
    A[Client needs ticket] --> B{udp_preference_limit}
    B -->|default| C[Try UDP :88]
    C -->|fail| D[Try TCP :88]
    C -->|ok| E[Got ticket]
    D --> E
    B -->|0| F[Use TCP :88 only]
    F --> E

Loading

Why it helps in this CI flake

The Ozone failure showed:

Caused by: java.net.PortUnreachableException
  at ... KdcComm.send ...

The Java client sent a UDP request to the KDC before UDP was reliably reachable, while the old KDC healthcheck only validated TCP with nc. So healthcheck could pass (TCP ✅) while Ozone login failed (UDP ❌).

Forcing TCP aligns client behavior with the healthcheck and avoids UDP startup races in Docker Compose.

Trade-offs

Aspect	Effect
Reliability in Docker CI	Better — avoids UDP race with KDC startup
Performance	Slightly more TCP overhead; negligible for CI smoke tests
Kerberos behavior	Does not disable Kerberos or change principals/keytabs — transport only
Scope	dev-support/ranger-docker only; production clusters use their own krb5.conf

Test plan

• plugins-docker-build passes on this PR (watch for multiple consecutive green runs)
• Local smoke: cd dev-support/ranger-docker && docker compose ... -f docker-compose.ranger-ozone.yml up -d — verify ozone-datanode, ozone-scm, ozone-om stay running
• Confirm docker inspect ranger-kdc reports healthy only after keytabs are provisioned
• Verify unrelated plugin containers (Hadoop, Hive, HBase, Kafka, Knox) still start normally

https://issues.apache.org/jira/browse/RANGER-5744