fix(ci): publish VM guest disks as compressed, all-or-nothing release assets - #38
Merged
Conversation
… assets The podman-vm publish path could leave a GitHub Release holding a checksum sidecar with no disk, and then wedge permanently. A GitHub Release asset must be under 2 GiB; the aarch64 raw disk is 2.3G. `gh release upload TAG disk disk.sha256` uploaded both in one invocation: the 105-byte checksum landed, the disk was rejected with "HTTP 422: Validation Failed ... size must be less than 2147483648", and the command exited non-zero with the orphan sidecar already published. Every later run then failed earlier still, with "asset under the same name already exists", because the idempotency guard only checked the disk name and never noticed the orphan. That is the entire content of release v25.08.15. Compress the disks with zstd before upload (originals kept, so the boot test, checksum gate and attestations still run against the real disks) and publish one architecture's set as a transaction: preflight every file for existence and the 2 GiB limit, roll back everything this run uploaded on any failure, delete and republish a partial set left by an earlier failed publish while never overwriting a complete one, and verify names and sizes on the release afterwards. A new verify-release job fails the run when the point-release tag is missing any asset for either architecture. Asset names now match what the launcher fetches: .raw.zst plus a .raw.sha256 for the decompressed disk, and a .raw.zst.sha256 for the download itself. Unblocks projectbluefin/donate-clanker#13. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Aug 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause (from the run logs, not inference)
Release
v25.08.15contains exactly one asset: a 105-bytedonate-clanker-vm-25.08.15-aarch64.raw.sha256. Run30684119054,
job
vm-guest / build (aarch64), stepPublish ...:A GitHub Release asset must be under 2 GiB. The exported aarch64 raw disk is
2.3G (
ls -lhin the same job).just publish-podman-vmuploaded the diskand its checksum in a single
gh release upload "$TAG" "$disk" "${disk}.sha256",which is not atomic: the 105-byte sidecar landed, the disk was rejected, and
the job failed with the orphan already published. Nothing was rolled back.
The release then wedged. Every later run failed one step earlier, most
recently run
30700620219:
because the idempotency guard checked only the disk name
(
gh release view ... | grep -qxF "$name"over"${raws[0]}"), so it nevernoticed the orphaned sidecar and re-attempted an upload that can only fail.
Two related findings:
donate-clanker-vm-25.08.14-x86_64.raw.zstonv25.08.14wasnot produced by CI. The release API shows
uploader: castrojo; theworkflow has never compressed anything. It is a hand-built workaround for
the same 2 GiB limit, which is why the workflow's
*.rawglobs nevermatched the published name.
vm-guest / build (x86_64)on run 30700620219 fails earlier, in theQEMU/Lima boot test (
did not receive an event with the running statusafter 300s). That is a separate guest defect, out of scope here; it
correctly blocks publication rather than half-publishing.
What changed
just compress-podman-vm(new) compresses both disks with zstd,--keep, sothe checksum gate, the boot test and both attestations still operate on the
real disks.
just publish-podman-vmis rewritten around one architecture'scomplete asset set.
How partial publication is now impossible, or at minimum unmissable
is uploaded. The failure mode that caused this bug is now a loud error
before the first byte, naming the file and its size.
trap ... ERRdeletes them again on any later failure. A failed runleaves the release exactly as it found it.
overwritten. A partial set is debris from a failed publish, not a
published artifact: the orphans are deleted and the full set republished.
This unwedges
v25.08.15automatically.name must be present at its expected byte size, or the run rolls back and
fails.
verify-releasejob(
needs: build,if: always()) fails the run when the tag is missing anyasset for either architecture. Publication stays inside each matrix leg,
so nothing is stranded; only the check is aggregate.
No
continue-on-erroranywhere. The checksum gate, the QEMU/Lima boot test,the SBOM and both
actions/atteststeps all still run; the attestationsubjects now additionally cover the
.zstassets that consumers download.Asset naming, reconciled
Per architecture, predictable from version and arch:
This is exactly what
projectbluefin/donate-clankeralready fetches(
just/61-donate-clanker.just: download.raw.zst,zstd -d, thensha256sum -cthe.raw.sha256), so the hand-uploadedv25.08.14layoutbecomes the CI-produced contract.
Verified vs inferred
Verified:
actionlint .github/workflows/*.yml- clean.shellcheckon both recipe bodies - clean.just validate- exit 0, element graph resolves.ghstub covering:fresh publish, re-run (skips, immutable), the real
v25.08.15orphansidecar state (repaired to a complete set), a mid-upload API failure
(rolled back to zero assets), an oversized asset (rejected in preflight,
zero assets uploaded), and a size-mismatched upload (caught by post-verify,
rolled back).
just compress-podman-vmexecuted on fixture files; the.zst.sha256manifests verify with
sha256sum -c.Inferred, not run: the multi-GB BuildStream export and the real GitHub upload
path.
just export-podman-vmbuilds a multi-GB disk in a container and wasnot run end to end here. The zstd level (
-12) is expected to land the 2.3Graw comfortably under 2 GiB, consistent with the 1.4G hand-compressed
v25.08.14asset; if a future disk ever exceeds the limit, preflight failsloudly rather than half-publishing.
Follow-up needed after merge
v25.08.15still needs a re-run ofBuild imagesonmain(no new tagrequired): the aarch64 leg will delete the orphan sidecar and publish the
complete set. A complete two-architecture release additionally needs the
x86_64 guest boot regression fixed - that is a separate issue and this PR
deliberately does not paper over it.
Unblocks projectbluefin/donate-clanker#13.