Skip to content

ci: add retry logic for Docker image pulls to prevent transient TLS timeout failures #382

Description

@millerjp

Symptoms

CI jobs intermittently fail with Docker Hub TLS handshake timeouts when pulling images:

net/http: TLS handshake timeout
Error response from daemon: Head "https://registry-1.docker.io/v2/hashicorp/vault/manifests/1.15": net/http: TLS handshake timeout

Observed on 3 jobs in run 23232779139: Docker Build, Python Data Contract & CSFLE Tests, BDD Tests (KMS + MySQL). All other 44 jobs passed.

Root Cause

No retry logic at any layer:

  1. composeUpWithProject() in tests/bdd/bdd_test.go (line ~3127) does a single docker compose up attempt — if the image pull fails, the entire test fails immediately
  2. CI workflow (.github/workflows/ci.yaml) has no retry or pre-pull steps for Docker images
  3. 13 images are pulled from Docker Hub by tag with no fallback, caching, or digest pinning

Suggested Improvements (prioritised)

P1: Add retry wrapper in bdd_test.go

Wrap composeUpWithProject() calls with exponential backoff retry (3 attempts, 1s/2s/4s). This is the single highest-impact change — it covers all BDD test Docker Compose invocations.

P2: Pre-pull critical images early in CI jobs

Add a step in .github/workflows/ci.yaml that pre-pulls the most-used images before test execution, isolating pull failures from test timeouts:

- name: Pre-pull Docker images
  run: |
    for img in postgres:15-alpine mysql:8.0 cassandra:5.0 hashicorp/vault:1.15 openbao/openbao:2.1.1; do
      docker pull "$img" || docker pull "$img" || echo "WARNING: failed to pre-pull $img"
    done

P3: Pin images to digests

Replace tag-based references (e.g., mysql:8.0) with digest-based references in Docker Compose files for deterministic, cache-friendly pulls.

Affected Files

  • tests/bdd/bdd_test.gocomposeUpWithProject() needs retry wrapper
  • .github/workflows/ci.yaml — add pre-pull step and/or retry on compose steps
  • tests/bdd/docker-compose*.yml — 13 images referenced by tag across 17 compose files

Metadata

Metadata

Assignees

No one assigned

    Labels

    buildCI and Build related tasks

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions