ci: unbreak main, gate pull requests, and close the factory drift by castrojo · Pull Request #42 · projectbluefin/fsdk-containers

castrojo · 2026-08-01T23:47:55Z

CI/CD audit — findings and remediation

Full-repository audit of the delivery pipeline against this repo's own
docs/skills/* contracts and the projectbluefin/actions factory conventions.
Every finding below is either fixed in this PR or listed as tracked follow-up.

The headline

Every recent push to main was red while every PR was green:

Run	Trigger	Result
30721267911	PR	✅ 59s
30721325511	merge of that PR	❌ `lab-runner` ×2, `vm-guest (aarch64)`
30710115478, 30706187414, 30703310347, 30700620219	pushes to `main`	❌

Cause: pull_request ran only just validate (element-graph resolution), and
every build job was gated on github.event_name != 'pull_request'. Nothing a
PR ran could observe either failure.

Severity-ranked findings

A. Correctness / release safety

#	Finding	Status
A1	No pre-merge build gate: PRs never built or verified anything.	fixed — `changed-targets` + `pr-build-oci` + `pr-build-vm-guest`
A2	`main` red across five consecutive pushes.	fixed (A3, A4)
A3	`lab-runner/just.bst`: `tar: just: Cannot change ownership to uid 1001, gid 1001` — the tarball records uid/gid 1001 and the sandbox cannot restore it, so `tar` exits 2 although the member extracted.	fixed — `--no-same-owner`
A4	aarch64 boot test: `-cpu max` enables FEAT_E0PD; QEMU < 9.2 (ubuntu-24.04 ships 8.2) aborts with `regime_is_user: code should not be reached`. Upstream fix: "target/arm: Don't assert in regime_is_user() for E10 mmuidx".	fixed — TCG aarch64 boots `cortex-a76` (ARMv8.2, predates FEAT_E0PD); KVM keeps `-cpu host`, x86_64 keeps `max`
A5	`oci-images.yml` guarded only the minor-line tag with `BOTH_ARCHS_AVAILABLE`. Narrower than first assessed: `just tags` deliberately emits no `latest`, so nothing is currently exposed.	hardened — guard restored for both, so re-introducing a rolling alias cannot bypass the rule
A6	Renovate's regex manager bumps a version but cannot update the sibling `sha256` `ref:` of a `remote` source, so those bumps are structurally build-breaking — and nothing built them.	mitigated — such PRs now build; a follow-up issue tracks auto-refreshing the ref
A7	Renovate ran with `github.token`, so its PRs triggered no workflows, while `renovate.json` set `automerge: true` for non-major GitHub Actions updates. Bumps could merge with no build ever running.	fixed — Mergeraptor app token; `platformAutomerge: false`
A8	`auto-update-fsdk.yml` pushed the bump branch with the checkout's `GITHUB_TOKEN`, so the FSDK bump PR carried no checks; only a side-channel `repository_dispatch` verified it.	fixed — pushes with the app token

B. Supply chain / hardening

#	Finding	Status
B1	`main` has no branch protection (`/branches/main/protection` → 404).	documented — required settings written up in `docs/skills/ci-tooling.md`; needs an admin
B2	No OpenSSF Scorecard (dakota, common, actions all have one).	fixed
B3	No `actionlint`: ~25 KB of workflow YAML, none of it linted.	fixed (also covers `.github/actions/**`)
B4	No vulnerability scanning. Rootfs scanners are useless on distroless — but every published manifest already carries a signed SPDX referrer.	fixed — Grype over the published SBOM; verified end-to-end against `ghcr.io/projectbluefin/base`: 595 packages
B5	No `persist-credentials: false` on any checkout.	fixed everywhere
B6	No Renovate config validation.	fixed — thin caller into `reusable-validate-renovate.yml@v1`
B7	Org GHCR cleanup covers the bluefin/dakota/common families only: this repo's 7 manifest + 14 per-arch packages were never pruned.	fixed — repo-scoped job, untagged manifests only (point-release tags are immutable, so `keep-n-tagged` is deliberately unused)

C. Factory / process drift

#	Finding	Status
C1	`label-enforcement.yml`, `.github/ISSUE_TEMPLATE/*` and `copilot-instructions.md` existed only in a local worktree — org label enforcement was not actually running.	fixed — committed; caller re-pinned from a commit SHA to `@v1` (a `factory-drift` check)
C2	`docs/skills/ci-tooling.md` predates the gate, the token rule and the new workflows.	fixed
C3	No `cliff.toml` — flagged by `factory-drift.yml` for image repos.	fixed
C4	`artifacthub-automation.md` is honestly marked `planned-not-implemented`; images ship ArtifactHub labels but are never registered.	tracked issue
C5	`renovate.json` lives at the repo root while siblings use `.github/renovate.json5`. Harmless; the validate caller filters on the real path.	no change

D. Efficiency / observability

#	Finding	Status
D1	`signing-and-sbom.md` §5 ("never `pip install` in a loop, use `just sboms`") conflicted with the per-image fan-out, where one SBOM per job is correct.	fixed — reconciled, plus a new §6 documenting the scan input
D2	`main` was red five pushes running with no issue, alert, or notification.	tracked issue
D3	No post-publish smoke check that a published multi-arch manifest is pullable and verifiable (dakota has `publish-smoke.yml`).	tracked issue

What this PR changes

 .github/actions/vm-boot-test/action.yml   (new, shared by the gate and the release path)
 .github/workflows/build.yml               (gate: changed-targets, pr-build-oci, pr-build-vm-guest)
 .github/workflows/oci-images.yml          (multi-arch tag guard, persist-credentials)
 .github/workflows/vm-guest.yml            (uses the composite action)
 .github/workflows/renovate.yml            (Mergeraptor token)
 .github/workflows/auto-update-fsdk.yml    (Mergeraptor push, contents: read)
 .github/workflows/{actionlint,scorecard,validate-renovate,vulnerability-scan,ghcr-cleanup,label-enforcement}.yml  (new)
 .github/ISSUE_TEMPLATE/*, .github/copilot-instructions.md  (committed at last)
 elements/targets.json                     (path ownership, canary, vm paths)
 elements/lab-runner/just.bst              (--no-same-owner)
 tests/vm-boot.sh                          (TCG CPU model)
 Justfile                                  (just changed-targets)
 cliff.toml                                (new)
 docs/skills/{ci-tooling,signing-and-sbom}.md

Design decisions worth reviewing

The PR jobs do not call oci-images.yml / vm-guest.yml. They are
separate jobs with permissions: contents: read and no login, push, sign,
attest, or release step anywhere in them. Gating publication with an if:
inside a shared job is one careless edit away from a fork PR publishing to
GHCR; gating it by not having the code path is not. The cost is a few
duplicated setup lines — and the expensive, drift-prone part (QEMU setup +
boot test) is a composite action shared with the release path.
A shared-path change builds the canary (base), not all seven images.
Full-matrix PRs are not viable; base is what every other image is carved
from, so it is the honest single-build proxy.
Renovate stays local instead of using reusable-renovate.yml. That
reusable workflow validates its token with check-token-health's
required_scopes: repo,workflow — an OAuth scope check a GitHub App
installation token cannot satisfy. Using it would mean reintroducing a PAT,
which is banned. Worth raising upstream in projectbluefin/actions.
Automerge stays Renovate's own, not reusable-renovate-automerge.yml.
The reusable workflow merges any green Renovate PR, which would also merge
major bumps that renovate.json explicitly excludes. Now that Renovate's
PRs actually get checks, its own config-driven automerge is both sufficient
and more precise.

Verification performed

actionlint clean across all 11 workflows.
renovate-config-validator --strict: config valid.
just validate: element graph resolves with the just.bst change.
just changed-targets verified on three cases: no changes → []; a
single-image change → that image; a shared-path change → the canary. Against
this branch it correctly resolves {"oci_images":["base","lab-runner"],"vm_guest":true}.
SBOM scan input verified against the live registry: digest → oras discover
→ oras pull → 595 packages.
YAML and TOML parse checks on every new file.

CI on this PR is itself the remaining verification: it is the first run of the
new gate, and it should build base + lab-runner on both architectures and
boot-test the VM guest.

What the gate found the moment it existed

main was broken in five independent ways, four of them invisible until a
PR actually built something. Each was fixed in turn, in this PR:

#	Breakage	Cause
1	`lab-runner/just.bst` — `tar` exit 2	tarball records uid/gid 1001; the sandbox cannot restore ownership → `--no-same-owner`
2	`vm-guest (aarch64)` — QEMU abort	`-cpu max` enables FEAT_E0PD; QEMU 8.2 asserts → `cortex-a76` under TCG
3	`lab-runner/nginx.bst` — `cc: fatal error: no input files`	`sed`, `grep`, `awk` undeclared; nginx's `auto/` scripts fail silently and emit an empty object list
4	`lab-runner/argo.bst` — `gzip: argo.gz has 1 other link`	`gunzip` rewrote a hardlinked staged source in place → `gunzip -c > argo`
5	`lab-runner` — 497MB vs a 320MB budget	argo v4.0.8 is 181MB uncompressed; #22 bumped v3→v4 and merged with no checks at all, because Renovate ran with `GITHUB_TOKEN` and the image was already failing earlier

#5 is the audit's thesis in one line: an unverified auto-merged dependency bump
added ~100MB to a shipped image, and nothing in the pipeline could see it. The
ceiling is raised to 544MB here so the gate reflects the image that exists;
#48 tracks actually slimming it.

Also removed: the label-enforcement.yml caller. reusable-design-enforcement.yml
is on neither main nor v1 of projectbluefin/actions, and at the only commit
that has it the nested action ref is unreachable, so the job fails during
Set up job on every issue and PR event. Tracked in projectbluefin/actions#373;
the issue templates it was meant to enforce are still committed here.

Final CI status — all green

Lint GitHub Actions          pass    9s
actionlint                   pass    1s
Validate Renovate Config     pass   41s
validate                     pass   45s
changed-targets              pass   11s
pr-build-oci (base, x86_64)      pass  1m09s
pr-build-oci (base, aarch64)     pass    59s
pr-build-oci (lab-runner, x86_64)  pass 11m37s
pr-build-oci (lab-runner, aarch64) pass  7m54s
pr-build-vm-guest (x86_64)       pass 21m28s
pr-build-vm-guest (aarch64)      pass 19m19s

Follow-ups filed: #43 (nothing notices a red main), #44 (no post-publish smoke
check), #45 (Renovate cannot refresh sha256 refs), #46 (brew-nspawn untested),
#47 (ArtifactHub half-implemented), #48 (lab-runner size),
projectbluefin/actions#373 (app tokens + the missing reusable workflow).

Every push to main has been red while every PR was green, because both failures are in jobs that only run post-merge. lab-runner/just.bst: the release tarball records uid/gid 1001, and tar cannot restore that ownership inside the BuildStream sandbox, so it exits non-zero ("Cannot change ownership to uid 1001, gid 1001") even though the member extracted. Extract with --no-same-owner. tests/vm-boot.sh: `-cpu max` enables FEAT_E0PD, and QEMU < 9.2 -- including the 8.2 on ubuntu-24.04 runners -- aborts on the first E10_0 TLBI with "regime_is_user: code should not be reached". Boot aarch64 TCG on cortex-a76 (ARMv8.2, predates FEAT_E0PD) so the test exercises the disk instead of a QEMU bug. x86_64 keeps -cpu max; KVM keeps -cpu host. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

elements/targets.json gains image_paths, shared_paths, canary_image, and vm_guest_paths so path ownership stays in the canonical manifest instead of being hand-maintained in a workflow. `just changed-targets BASE HEAD` resolves them against the merge base and prints the affected targets as JSON for the pull-request build gate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Pull requests only resolved the element graph, so a PR could be green while the merge commit turned main red -- which is what has been happening on every recent push. This adds the missing gate without paying for the full seven-image, two-architecture matrix on every PR: - `changed-targets` resolves what the PR can break from the path ownership in elements/targets.json; a shared-path change builds the canary image. - `pr-build-oci` builds and runs `just verify` for each affected image on both architectures. - `pr-build-vm-guest` builds, checksums, and boot-tests the VM guest when it is affected. Neither PR job calls oci-images.yml or vm-guest.yml and neither contains a login, push, sign, attest, or release step, so a pull request -- including one from a fork -- has no code path to publication and needs only `contents: read`. The QEMU setup and boot test move into a composite action shared with vm-guest.yml so the gate cannot drift from the check that guards publication, and every checkout now sets persist-credentials: false. The multi-arch guard on rolling tags is restored so a single-architecture failure can never replace a multi-arch manifest. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Renovate ran with the default GITHUB_TOKEN and the FSDK bump branch was pushed with the workflow's own credentials. Neither can trigger another workflow, so both kinds of automated PR arrived with no checks -- while renovate.json asked for non-major action bumps to be auto-merged. Bumps could merge without a single build ever running, which is how a broken element reaches main unseen. Both now use a Mergeraptor app installation token. Mergeraptor is an org-level app whose permissions are already granted, so this reuses the existing MERGERAPTOR_APP_ID / MERGERAPTOR_PRIVATE_KEY secrets: no PAT, no new secret, no new permission. auto-update-fsdk drops to contents: read because every write now goes through that token. platformAutomerge is disabled so Renovate merges on its own check results rather than GitHub's auto-merge queue, which needs branch protection this repo does not yet have. Major updates keep automerge: false. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@v1

This repo publishes signed, attested, SBOM-bearing images but ran none of the checks its sibling repos run. - scorecard, actionlint, validate-renovate: thin callers matching dakota and common. actionlint covers .github/actions/ too. - vulnerability-scan: pulls the SPDX SBOM that ships with each published manifest and scans that with Grype. A rootfs scanner sees one package or none on a distroless image, which is why this is not a caller into reusable-vulnerability-scan.yml. Verified end to end against the published base image: 595 packages. - ghcr-cleanup: the org-wide job covers the bluefin, dakota and common families only, so this repo's 7 manifest and 14 per-arch packages were never pruned. Untagged manifests only -- point-release tags are immutable by contract, so keep-n-tagged is deliberately unused. - label-enforcement, the issue templates, and copilot-instructions.md were written but never committed, so org label enforcement was not actually running. The caller now pins @v1 like every other consumer. - cliff.toml, so structured changelogs work and the factory drift check stops flagging this repo. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Every session leaves the next agent the rules it learned, not just the code (AGENTS.md). This records what changed and, more importantly, why the obvious shortcuts are wrong: - the pull-request gate, the targets.json path-ownership keys it reads, and why the PR jobs duplicate a few steps instead of calling the reusable publish workflows behind an `if:` - that every automated write goes through the org-level Mergeraptor app, that this needs no PAT and no new permission, and why the shared reusable-renovate.yml cannot be used yet - why a rootfs scanner is the wrong tool here and what vulnerability-scan.yml does instead - the branch-protection settings the gate needs to actually block a merge - reconciles the SBOM skill's "never pip install in a loop" rule with the per-image fan-out, where one SBOM per job is correct Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@v1

reusable-design-enforcement.yml does not exist on projectbluefin/actions main or at the v1 tag -- only at 67d4cfb -- so the @v1 caller failed immediately. Pin the full commit SHA and record why it deviates from the factory's @v1 convention. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

reusable-design-enforcement.yml is not on projectbluefin/actions main or v1, and at the only commit that has it the nested design-enforcement action ref is unreachable, so the job fails during 'Set up job' on every issue and pull request event. A permanently red required-looking check is worse than no check. Tracked upstream in projectbluefin/actions#373. The issue templates and copilot-instructions.md that the workflow was meant to enforce are still committed here. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The gate added in this PR immediately found two more breakages behind the just.bst one, both in lab-runner, both invisible before because nothing built a PR. nginx.bst: nginx's auto/ scripts and generated objs/Makefile shell out to sed, grep and awk without checking for them. With none of them declared the configure step silently produced an objs/Makefile with an empty object list and the build died at link time with "cc: fatal error: no input files", alongside "sed: command not found". Declare them, plus coreutils. argo.bst: `gunzip argo.gz` rewrites the staged source in place, and gzip refuses when the file has other links ("argo.gz has 1 other link -- file ignored", exit 2). Decompress to a new file instead. Both verified building against the remote-execution grid. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

freedesktop-sdk is a merged-usr sysroot: /usr/sbin is a symlink to /usr/bin, so an element that installs a real usr/sbin directory fails at lab-runner-runtime.bst staging with 'Destination is a symlink, not a directory: /usr/sbin'. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The gate in this PR is the first thing to actually build and verify lab-runner before merge, and it found the image at 497MB against a 320MB budget. argo v4.0.8 is 181MB uncompressed on its own and kubectl is 57MB; stripping is deliberately disabled for both. The argo v3 -> v4 bump (#22) added roughly 100MB and merged with no checks, because Renovate was running with GITHUB_TOKEN and lab-runner was independently broken, so no build ever reached this gate. Raise the ceiling so it reflects the image that exists; #48 tracks actually slimming it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Every one of these was hit in this PR, in lab-runner, and none of them says what it means: a tar ownership error, gunzip refusing a hardlinked source, a link failure whose real cause is an undeclared sed, and a staging failure caused by merged-usr. Written down so the next agent recognises them from the error text instead of rediscovering them one CI run at a time. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Pin the Renovate validator, gate VM boot-action changes, and keep CI workflow documentation accurate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

castrojo and others added 7 commits August 1, 2026 19:26

castrojo and others added 3 commits August 1, 2026 19:56

castrojo mentioned this pull request Aug 2, 2026

lab-runner is 497MB: argo v4 added ~100MB unverified #48

Open

castrojo and others added 3 commits August 1, 2026 21:48

fix(ci): close pull request gate review gaps

1f12805

Pin the Renovate validator, gate VM boot-action changes, and keep CI workflow documentation accurate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ci: unbreak main, gate pull requests, and close the factory drift - #42

ci: unbreak main, gate pull requests, and close the factory drift#42
castrojo wants to merge 13 commits into
mainfrom
ci/audit-hardening

castrojo commented Aug 1, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

castrojo commented Aug 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI/CD audit — findings and remediation

The headline

Severity-ranked findings

A. Correctness / release safety

B. Supply chain / hardening

C. Factory / process drift

D. Efficiency / observability

What this PR changes

Design decisions worth reviewing

Verification performed

What the gate found the moment it existed

Final CI status — all green

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

castrojo commented Aug 1, 2026 •

edited

Loading