Skip to content

install-ate.sh: --deploy-ate-system never re-applies CRDs/RBAC on upgrade #697

Description

Summary

deploy_ate_system gates manifests/ate-install/generated/ behind ensure_crds, which returns early when the three CRDs already exist. On any upgrade of an existing cluster that skips the whole directory — and since role.yaml lives there and no other deploy path applies it, both CRD schemas and every ClusterRole stay frozen at first-install state.

A controller image that needs a newly added permission then deadlocks at startup: its informer cache cannot list the resource, so reconciliation never runs, while the rollout itself reports success.

Observed on a real cluster

kind cluster installed 4 days ago, then upgraded via install-ate-kind.sh --deploy-ate-system:

  • ate-controller (new image) logged
    failed to list *v1.NetworkPolicy: ... User "system:serviceaccount:ate-system:ate-controller" cannot list resource "networkpolicies"
    and stopped reconciling entirely — a deleted WorkerPool Deployment was never recreated, with no events anywhere.
  • After manually applying manifests/ate-install/generated/role.yaml and restarting the controller, reconciliation resumed immediately.
  • With the fix, the same upgrade run printed actortemplates.ate.dev configured / workerpools.ate.dev configured — the CRD schemas had also silently drifted.

Reproduction

  1. Install: ./hack/install-ate-kind.sh --deploy-ate-system
  2. Advance the tree to any revision whose controller requires a new RBAC rule (or edit the live ClusterRole to drop one).
  3. Upgrade with the same command: rollout reports success, deploy_crds never runs, controller deadlocks on informer start.

Suggested fix

Call deploy_crds unconditionally from deploy_ate_systemkubectl apply is idempotent, so re-applying on every install/upgrade is free. ensure_crds remains correct for the demo scripts and per-component deploy flags, where the semantic really is "make sure they exist".

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions