Skip to content

fix(ci): publish VM guest disks as compressed, all-or-nothing release assets - #38

Merged
castrojo merged 1 commit into
mainfrom
fix/vm-guest-publish
Aug 1, 2026
Merged

fix(ci): publish VM guest disks as compressed, all-or-nothing release assets#38
castrojo merged 1 commit into
mainfrom
fix/vm-guest-publish

Conversation

@castrojo

@castrojo castrojo commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Root cause (from the run logs, not inference)

Release v25.08.15 contains exactly one asset: a 105-byte
donate-clanker-vm-25.08.15-aarch64.raw.sha256. Run
30684119054,
job vm-guest / build (aarch64), step Publish ...:

==> creating release v25.08.15 (none exists yet)
HTTP 422: Validation Failed (https://uploads.github.com/repos/projectbluefin/fsdk-containers/releases/363432170/assets?label=&name=donate-clanker-vm-25.08.15-aarch64.raw)
size must be less than 2147483648
error: recipe `publish-podman-vm` failed with exit code 1

A GitHub Release asset must be under 2 GiB. The exported aarch64 raw disk is
2.3G (ls -lh in the same job). just publish-podman-vm uploaded the disk
and its checksum in a single gh release upload "$TAG" "$disk" "${disk}.sha256",
which is not atomic: the 105-byte sidecar landed, the disk was rejected, and
the job failed with the orphan already published. Nothing was rolled back.

The release then wedged. Every later run failed one step earlier, most
recently run
30700620219:

asset under the same name already exists: [donate-clanker-vm-25.08.15-aarch64.raw.sha256]
error: recipe `publish-podman-vm` failed with exit code 1

because the idempotency guard checked only the disk name
(gh release view ... | grep -qxF "$name" over "${raws[0]}"), so it never
noticed the orphaned sidecar and re-attempted an upload that can only fail.

Two related findings:

  • The donate-clanker-vm-25.08.14-x86_64.raw.zst on v25.08.14 was
    not produced by CI. The release API shows uploader: castrojo; the
    workflow has never compressed anything. It is a hand-built workaround for
    the same 2 GiB limit, which is why the workflow's *.raw globs never
    matched the published name.
  • vm-guest / build (x86_64) on run 30700620219 fails earlier, in the
    QEMU/Lima boot test (did not receive an event with the running status
    after 300s). That is a separate guest defect, out of scope here; it
    correctly blocks publication rather than half-publishing.

What changed

just compress-podman-vm (new) compresses both disks with zstd, --keep, so
the checksum gate, the boot test and both attestations still operate on the
real disks. just publish-podman-vm is rewritten around one architecture's
complete asset set.

How partial publication is now impossible, or at minimum unmissable

  1. Preflight. Every asset must exist and be under 2 GiB before anything
    is uploaded. The failure mode that caused this bug is now a loud error
    before the first byte, naming the file and its size.
  2. Rollback. Every asset uploaded by an invocation is recorded, and a
    trap ... ERR deletes them again on any later failure. A failed run
    leaves the release exactly as it found it.
  3. Repair without weakening immutability. A complete set is never
    overwritten. A partial set is debris from a failed publish, not a
    published artifact: the orphans are deleted and the full set republished.
    This unwedges v25.08.15 automatically.
  4. Post-verify. The release is re-read after upload and every expected
    name must be present at its expected byte size, or the run rolls back and
    fails.
  5. Cross-architecture completeness. A new verify-release job
    (needs: build, if: always()) fails the run when the tag is missing any
    asset for either architecture. Publication stays inside each matrix leg,
    so nothing is stranded; only the check is aggregate.

No continue-on-error anywhere. The checksum gate, the QEMU/Lima boot test,
the SBOM and both actions/attest steps all still run; the attestation
subjects now additionally cover the .zst assets that consumers download.

Asset naming, reconciled

Per architecture, predictable from version and arch:

donate-clanker-vm-<version>-<arch>.raw.zst          <- the download
donate-clanker-vm-<version>-<arch>.raw.zst.sha256   <- verifies the download
donate-clanker-vm-<version>-<arch>.raw.sha256       <- verifies the disk after decompression
donate-clanker-vm-<version>-<arch>.qcow2.zst
donate-clanker-vm-<version>-<arch>.qcow2.zst.sha256
donate-clanker-vm-<version>-<arch>.qcow2.sha256
podman-vm-<arch>.spdx.json

This is exactly what projectbluefin/donate-clanker already fetches
(just/61-donate-clanker.just: download .raw.zst, zstd -d, then
sha256sum -c the .raw.sha256), so the hand-uploaded v25.08.14 layout
becomes the CI-produced contract.

Verified vs inferred

Verified:

  • actionlint .github/workflows/*.yml - clean.
  • shellcheck on both recipe bodies - clean.
  • just validate - exit 0, element graph resolves.
  • The rewritten publish recipe was executed against a gh stub covering:
    fresh publish, re-run (skips, immutable), the real v25.08.15 orphan
    sidecar state (repaired to a complete set), a mid-upload API failure
    (rolled back to zero assets), an oversized asset (rejected in preflight,
    zero assets uploaded), and a size-mismatched upload (caught by post-verify,
    rolled back).
  • just compress-podman-vm executed on fixture files; the .zst.sha256
    manifests verify with sha256sum -c.

Inferred, not run: the multi-GB BuildStream export and the real GitHub upload
path. just export-podman-vm builds a multi-GB disk in a container and was
not run end to end here. The zstd level (-12) is expected to land the 2.3G
raw comfortably under 2 GiB, consistent with the 1.4G hand-compressed
v25.08.14 asset; if a future disk ever exceeds the limit, preflight fails
loudly rather than half-publishing.

Follow-up needed after merge

v25.08.15 still needs a re-run of Build images on main (no new tag
required): the aarch64 leg will delete the orphan sidecar and publish the
complete set. A complete two-architecture release additionally needs the
x86_64 guest boot regression fixed - that is a separate issue and this PR
deliberately does not paper over it.

Unblocks projectbluefin/donate-clanker#13.

… assets

The podman-vm publish path could leave a GitHub Release holding a checksum
sidecar with no disk, and then wedge permanently.

A GitHub Release asset must be under 2 GiB; the aarch64 raw disk is 2.3G.
`gh release upload TAG disk disk.sha256` uploaded both in one invocation:
the 105-byte checksum landed, the disk was rejected with
"HTTP 422: Validation Failed ... size must be less than 2147483648", and
the command exited non-zero with the orphan sidecar already published.
Every later run then failed earlier still, with "asset under the same name
already exists", because the idempotency guard only checked the disk name
and never noticed the orphan. That is the entire content of release
v25.08.15.

Compress the disks with zstd before upload (originals kept, so the boot
test, checksum gate and attestations still run against the real disks) and
publish one architecture's set as a transaction: preflight every file for
existence and the 2 GiB limit, roll back everything this run uploaded on any
failure, delete and republish a partial set left by an earlier failed
publish while never overwriting a complete one, and verify names and sizes
on the release afterwards. A new verify-release job fails the run when the
point-release tag is missing any asset for either architecture.

Asset names now match what the launcher fetches: .raw.zst plus a .raw.sha256
for the decompressed disk, and a .raw.zst.sha256 for the download itself.

Unblocks projectbluefin/donate-clanker#13.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@castrojo
castrojo merged commit ecf0e35 into main Aug 1, 2026
5 checks passed
@castrojo
castrojo deleted the fix/vm-guest-publish branch August 1, 2026 14:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant