Report graceful agent stops to Buildkite - #4270
Conversation
Report signal-driven graceful shutdowns by default, producing STOPPING while jobs drain and STOPPED after disconnect. Keep reporting best-effort and allow opting out with BUILDKITE_AGENT_REPORT_GRACEFUL_STOP=false.
![buildsworth-bk-app[bot]](https://avatars.githubusercontent.com/in/3010600?s=60&v=4){kind=small}
There was a problem hiding this comment.
The added control-plane call and synchronization alter the core worker shutdown lifecycle, so this needs human review under the current approval policy. I found no code issues to call out.
Want to dig deeper?
Paste this into your agent to explore the findings from this review's Buildkite build:
Download the buildsworth logs from build 20918, then answer my questions about the findings.
Install the reading-buildsworth-logs skill to run this.
About buildsworth
Model: gpt-5.6-sol with xhigh thinking.
How to request a review: Comment @buildsworth-bk review on the PR, or request buildsworth-bk as a reviewer.
Risk labels (how buildsworth classifies risk) — buildsworth classifies risk itself from the diff. Unless repository policy already allows L2 approval, grant it by mentioning @buildsworth-bk (see approval ceiling and L2 approval grant):
- L1 — Low risk (dep bumps, docs/copy, lockfiles, small presentational fixes). buildsworth may approve by default.
- L2 — Standard risk (new UI, additive API fields, refactors). Approved only when repository policy or a verified grant allows it; otherwise comment-only.
- L3 — High risk (auth, migrations, payments, secrets, perf-critical paths). Human review always required.
There was a problem hiding this comment.
The added control-plane call and synchronization alter the core worker shutdown lifecycle, so this still needs human review under the current L1 approval policy. I found no code issues to call out.
Want to dig deeper?
Paste this into your agent to explore the findings from this review's Buildkite build:
Download the buildsworth logs from build 22034, then answer my questions about the findings.
Install the reading-buildsworth-logs skill to run this.
1 participant
Description
Graceful signal shutdowns currently drain the worker locally, but the control plane shows it as
CONNECTEDuntil it disconnects.This reports a non-forced
/stopusing the worker's existing session, giving usSTOPPINGwhile jobs drain andSTOPPEDafter disconnect. Reporting is best-effort and limited to five seconds.It's enabled by default and can be disabled with
BUILDKITE_AGENT_REPORT_GRACEFUL_STOP=false. Ungraceful shutdown paths, including standard agent-stack-k8s termination, remain unchanged.I considered handling this through the scaler or Stacks API, but that would need broader credentials and agent-to-instance mapping. The worker already has everything it needs.
Context
buildkite/elastic-ci-stack-for-aws#1861
Changes
--report-graceful-stopflag and environment variable.Testing
go test ./...). Buildkite employees may check this if the pipeline has run automatically.go tool gofumpt -extra -w .)Affiliation (optional, external contributors)
Also tested the lifecycle on Elastic Stack with three workers: all moved to
STOPPINGwhile their jobs continued, then reachedSTOPPEDafter finishing.Disclosures / Credits