Symptoms
CI jobs intermittently fail with Docker Hub TLS handshake timeouts when pulling images:
net/http: TLS handshake timeout
Error response from daemon: Head "https://registry-1.docker.io/v2/hashicorp/vault/manifests/1.15": net/http: TLS handshake timeout
Observed on 3 jobs in run 23232779139: Docker Build, Python Data Contract & CSFLE Tests, BDD Tests (KMS + MySQL). All other 44 jobs passed.
Root Cause
No retry logic at any layer:
composeUpWithProject() in tests/bdd/bdd_test.go (line ~3127) does a single docker compose up attempt — if the image pull fails, the entire test fails immediately
- CI workflow (
.github/workflows/ci.yaml) has no retry or pre-pull steps for Docker images
- 13 images are pulled from Docker Hub by tag with no fallback, caching, or digest pinning
Suggested Improvements (prioritised)
P1: Add retry wrapper in bdd_test.go
Wrap composeUpWithProject() calls with exponential backoff retry (3 attempts, 1s/2s/4s). This is the single highest-impact change — it covers all BDD test Docker Compose invocations.
P2: Pre-pull critical images early in CI jobs
Add a step in .github/workflows/ci.yaml that pre-pulls the most-used images before test execution, isolating pull failures from test timeouts:
- name: Pre-pull Docker images
run: |
for img in postgres:15-alpine mysql:8.0 cassandra:5.0 hashicorp/vault:1.15 openbao/openbao:2.1.1; do
docker pull "$img" || docker pull "$img" || echo "WARNING: failed to pre-pull $img"
done
P3: Pin images to digests
Replace tag-based references (e.g., mysql:8.0) with digest-based references in Docker Compose files for deterministic, cache-friendly pulls.
Affected Files
tests/bdd/bdd_test.go — composeUpWithProject() needs retry wrapper
.github/workflows/ci.yaml — add pre-pull step and/or retry on compose steps
tests/bdd/docker-compose*.yml — 13 images referenced by tag across 17 compose files
Symptoms
CI jobs intermittently fail with Docker Hub TLS handshake timeouts when pulling images:
Observed on 3 jobs in run 23232779139: Docker Build, Python Data Contract & CSFLE Tests, BDD Tests (KMS + MySQL). All other 44 jobs passed.
Root Cause
No retry logic at any layer:
composeUpWithProject()intests/bdd/bdd_test.go(line ~3127) does a singledocker compose upattempt — if the image pull fails, the entire test fails immediately.github/workflows/ci.yaml) has no retry or pre-pull steps for Docker imagesSuggested Improvements (prioritised)
P1: Add retry wrapper in
bdd_test.goWrap
composeUpWithProject()calls with exponential backoff retry (3 attempts, 1s/2s/4s). This is the single highest-impact change — it covers all BDD test Docker Compose invocations.P2: Pre-pull critical images early in CI jobs
Add a step in
.github/workflows/ci.yamlthat pre-pulls the most-used images before test execution, isolating pull failures from test timeouts:P3: Pin images to digests
Replace tag-based references (e.g.,
mysql:8.0) with digest-based references in Docker Compose files for deterministic, cache-friendly pulls.Affected Files
tests/bdd/bdd_test.go—composeUpWithProject()needs retry wrapper.github/workflows/ci.yaml— add pre-pull step and/or retry on compose stepstests/bdd/docker-compose*.yml— 13 images referenced by tag across 17 compose files