Build E2E binary ahead of release matrix

[lox]

Description

Draft comparison for #4213.

Build the release Linux/amd64 artifact in one dedicated step so E2E can start without waiting for the other 21 release targets. Leave the existing release matrix and its downstream fan-in intact rather than splitting it into three dependency cohorts.

The Linux/amd64 target is skipped in the matrix, so this does not duplicate work or artifacts. E2E continues to test the exact artifact later consumed by release assembly.

Comparison goal

Determine whether this simpler graph retains the critical-path improvement measured in #4213:

• Reduce CI binary build fan-in #4213: three binary matrix keys and consumer-specific dependencies
• This draft: one dedicated Linux/amd64 step plus the otherwise unchanged matrix

Both branches persist E2E Go compilation/module caches and use -count=1, so cached test results cannot affect the comparison.

Measured comparison

Draft build 13874 passed in 8.86 minutes, versus 8.28 minutes for #4213's similarly queue-constrained final-code build 13872. This single comparison was 0.58 minutes / 7% slower, but total duration is queue-sensitive.

The simpler graph did start E2E substantially earlier:

Timing	Simpler draft #13874	Cohort PR #13872
Fast/core artifact ready	1.98m	3.39m
All binaries ready	3.90m	5.53m
E2E finished	5.40m	8.27m
Non-E2E release-consumer tail	8.85m	6.85m
Overall build	8.86m	8.28m

The dedicated artifact removed 1.92m of E2E fan-in, but unchanged release consumers all became runnable together after 3.92m and then queued for roughly 2-3.2 minutes. RPM packaging became the final job at 8.85m. The cohort design staged Docker/package work earlier, keeping its non-E2E tail two minutes shorter under comparable contention.

The draft still produced all 22 binaries and all 22 attestations, and every pipeline consumer passed. Its source diff is 36 additions / 4 deletions, compared with #4213's 71 additions / 13 deletions.

Conclusion: this design is materially simpler and gives much earlier E2E feedback, but this run did not preserve #4213's overall wall-clock result under contention.

Changes

• Extract Linux/amd64 from the release matrix into build-linux-amd64-binary.
• Make E2E and the amd64-only bk CLI test depend directly on that artifact.
• Make full-release consumers depend on both the dedicated artifact and remaining matrix.
• Persist Go build/module caches for E2E while disabling Go test-result reuse.

Tradeoffs

• Docker and package jobs still wait for the full release matrix, unlike Reduce CI binary build fan-in #4213.
• The release matrix now uses Buildkite's maximum of 12 adjustments.
• Only the Linux/amd64 attestation records a new step key; filenames and subjects remain unchanged.

Testing

• Tests have run locally (with go test ./...). Buildkite employees may check this if the pipeline has run automatically.
• Code is formatted (with go tool gofumpt -extra -w .)

No Go source changed. Validation performed:

• Parsed the pipeline with the repository's own uploader in dry-run mode with parse warnings rejected.
• Expanded the graph and verified exactly 22 unique producing targets, 22 unique attestation filenames, three intentional skips, and 12 matrix adjustments.
• Parsed both changed YAML files.
• Ran bash -n .buildkite/steps/e2e-tests.sh and git diff --check.

Buildkite build 13874 passed and complete artifact/release validation is recorded above.

Disclosures / Credits

Amp implemented and validated this comparison after analyzing the measured graph from #4213. OpenAI's Oracle reviewed the shared cache correctness constraints in #4213.