Add lazy Namespace storage client by buildkate · Pull Request #4257 · buildkite/agent

buildkate · 2026-08-24T01:10:15Z

Description

Replace nsc artifact subprocesses with the Namespace Storage API, removing the runtime dependency on the NSC CLI while preserving cache expiry, misses, metrics, and integrity checks.

Context

A-1435
Plan

Changes

Use direct API uploads, streamed downloads with transient retries, and best-effort 24-hour expiry refreshes.
Share one lazy Namespace client per cache command and close it after workers finish.
Exclude Namespace's unlicensed generated modules from acknowledgements generation (they confirmed they use the same license as in their repo); but it is not publish with the modules.

buildkite-agent cache save/restore
                 │
                 ▼
       Load cache configuration
                 │
                 ▼
      Resolve requested cache names
                 │
                 ▼
       Start cache worker goroutines
          ┌──────┼──────┐
          ▼      ▼      ▼
       Cache A Cache B Cache C
          └──────┼──────┘
                 ▼
        Wait for every worker
                 │
                 ▼
             Command exits

Testing

Tests have run locally (with go test ./...).
Ran Namespace integration test locally which makes real calls to Namespace
Code is formatted (with go tool gofumpt -extra -w .).

Disclosures / Credits

Implemented with Amp.

socket-security · 2026-08-24T01:10:35Z

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	golang/buf.build/gen/go/namespace/cloud/protocolbuffers/go@v1.36.12-20260820164744-a1973bcf4d87.1
	golang/namespacelabs.dev/integrations@v0.0.11-0.20260508113815-6a8135624a35

View full report

buildkate · 2026-08-24T06:38:48Z

 	Version   int    `json:"version"`          // Metadata schema version
 }

+func validateFilePath(filePath string) error {


This was moved unchanged from nsc.go as it's not required by the API but still used by the local file store for validation.

buildkate · 2026-08-24T06:40:50Z

 	"testing"
 )

+func TestValidateFilePath(t *testing.T) {


similarly to file.go, all these tests were just moved from nsc_test.go to here.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 948571aaff

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

buildsworth-bk-app

The direct download path drops the CLI downloader's retry behavior, leaving transient stream failures to abort cache restores. Details are inline.

Want to dig deeper?

Paste this into your agent to explore the findings from this review's Buildkite build:

Download the buildsworth logs from build 20459, then answer my questions about the findings.

Install the reading-buildsworth-logs skill to run this.

About buildsworth

Model: gpt-5.6-sol with xhigh thinking.

How to request a review: Comment @buildsworth-bk review on the PR, or request buildsworth-bk as a reviewer.

Risk labels (how buildsworth classifies risk) — buildsworth classifies risk itself from the diff. Unless repository policy already allows L2 approval, grant it by mentioning @buildsworth-bk (see approval ceiling and L2 approval grant):

L1 — Low risk (dep bumps, docs/copy, lockfiles, small presentational fixes). buildsworth may approve by default.
L2 — Standard risk (new UI, additive API fields, refactors). Approved only when repository policy or a verified grant allows it; otherwise comment-only.
L3 — High risk (auth, migrations, payments, secrets, perf-critical paths). Human review always required.

buildsworth-bk-app

The previous download-retry blocker is still open; the latest revision only renames the Namespace client types and fields.

Want to dig deeper?

Paste this into your agent to explore the findings from this review's Buildkite build:

Download the buildsworth logs from build 20467, then answer my questions about the findings.

Install the reading-buildsworth-logs skill to run this.

buildkate · 2026-08-25T00:13:37Z

@buildsworth-bk review

buildkate · 2026-08-25T00:13:37Z

@codex review

Head: a9e8c09

chatgpt-codex-connector · 2026-08-25T00:16:54Z

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: a9e8c096b1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

buildsworth-bk-app

The retry blocker remains open for one transient stream-failure class; details are in the existing thread. I found no other new concerns in this pass.

Want to dig deeper?

Paste this into your agent to explore the findings from this review's Buildkite build:

Download the buildsworth logs from build 20481, then answer my questions about the findings.

Install the reading-buildsworth-logs skill to run this.

buildkate · 2026-08-25T00:19:57Z

@buildsworth-bk review

buildkate · 2026-08-25T00:20:05Z

@codex review

Head: fd7dbb6

buildkate · 2026-08-25T00:27:02Z

@buildsworth-bk review

buildkate · 2026-08-25T00:29:03Z

@buildsworth-bk review

buildkate · 2026-08-25T00:29:04Z

@codex review

Head: 8986893

chatgpt-codex-connector · 2026-08-25T00:32:13Z

Codex Review: Didn't find any major issues. Swish!

Reviewed commit: 8986893d24

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

buildsworth-bk-app

The HTTP/2 reset path is fixed, but one part of the existing download-retry blocker remains open; details are in its thread. I found no other new concerns in this pass.

Want to dig deeper?

Paste this into your agent to explore the findings from this review's Buildkite build:

Download the buildsworth logs from build 20502, then answer my questions about the findings.

Install the reading-buildsworth-logs skill to run this.

buildkate · 2026-08-25T00:44:08Z

@buildsworth-bk review

buildkate · 2026-08-25T00:44:09Z

@codex review

Head: c2d4afc

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c2d4afccac

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Previous concerns have been addressed.

buildsworth-bk-app

The previous download-retry concern is fixed. The signed-download HTTP 404 miss-mapping thread is blocking this revision.

Want to dig deeper?

Paste this into your agent to explore the findings from this review's Buildkite build:

Download the buildsworth logs from build 20522, then answer my questions about the findings.

Install the reading-buildsworth-logs skill to run this.

buildkate · 2026-08-25T01:19:47Z

@buildsworth-bk review

buildkate · 2026-08-25T01:19:48Z

@codex review

Head: 72a965c

chatgpt-codex-connector · 2026-08-25T01:23:38Z

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: 72a965c820

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Previous concerns have been addressed.

buildsworth-bk-app

The signed-download HTTP 404 mapping now addresses the previous blocker, and I found no new concerns. This cache transport and shared client-lifecycle change is above the L1 approval ceiling, so I’m leaving this as a comment.

Want to dig deeper?

Paste this into your agent to explore the findings from this review's Buildkite build:

Download the buildsworth logs from build 20557, then answer my questions about the findings.

Install the reading-buildsworth-logs skill to run this.

jamiemonserrate

Have you tested back portability? I'm just curious if our existing pipelines (bk/bk) will continue to work, once we move to this. It should - but just curious if you tested.
Any reason you opted to go for a single stream download? This is going to be significantly slower that the exisiting nsc CLI. And it seems we lose all the chunk based retry, and the exponential backoff that nsc CLI gave us. Any reason for dropping this?

buildkate · 2026-08-26T01:00:52Z

Have you tested back portability? I'm just curious if our existing pipelines (bk/bk) will continue to work, once we move to this. It should - but just curious if you tested.

Any reason you opted to go for a single stream download? This is going to be significantly slower that the exisiting nsc CLI. And it seems we lose all the chunk based retry, and the exponential backoff that nsc CLI gave us. Any reason for dropping this?

@jamiemonserrate

re testing: will do testing before merging
re single stream download vs CLI behavior: I switched to using the Namespace SDK which has no built in helper to do the chunking and retry that the CLI had. So admittedly the this differs to the CLI implementation.

These are the three options I see:

Use ResolveArtifactStream

Lowest implementation effort.
Removes the CLI dependency.
Loses parallel throughput and chunk-level retries.
Whole-file retries can retransmit large amounts of data.

Implement ranged downloads using ResolveArtifact

Removes the CLI dependency while preserving performance and resilience.
Require a lot of time implementation, testing, and validation + additional complexity
We own ongoing downloader maintenance

Keep using the CLI

Preserves existing proven download behavior.
Lowest engineering effort.
Does not solve A-1435: containerized jobs without nsc continue to fail.
Retains subprocess and CLI-version dependencies.

Given that I'll be doing regression testing, if the results are tolerable with the current implementation (option 1), I will go with that.

Wdyt?

jamiemonserrate · 2026-08-26T01:08:51Z

Have you tested back portability? I'm just curious if our existing pipelines (bk/bk) will continue to work, once we move to this. It should - but just curious if you tested.

Any reason you opted to go for a single stream download? This is going to be significantly slower that the exisiting nsc CLI. And it seems we lose all the chunk based retry, and the exponential backoff that nsc CLI gave us. Any reason for dropping this?

@jamiemonserrate

re testing: will do testing before merging

re single stream download vs CLI behavior: I switched to using the Namespace SDK which has no built in helper to do the chunking and retry that the CLI had. So admittedly the this differs to the CLI implementation.

These are the three options I see:

Use ResolveArtifactStream

Lowest implementation effort.

Removes the CLI dependency.

Loses parallel throughput and chunk-level retries.

Whole-file retries can retransmit large amounts of data.

Implement ranged downloads using ResolveArtifact

Removes the CLI dependency while preserving performance and resilience.

Require a lot of time implementation, testing, and validation + additional complexity

We own ongoing downloader maintenance

Keep using the CLI

Preserves existing proven download behavior.

Lowest engineering effort.

Does not solve A-1435: containerized jobs without nsc continue to fail.

Retains subprocess and CLI-version dependencies.

Given that I'll be doing regression testing, if the results are tolerable with the current implementation (option 1), I will go with that.

Wdyt?

Yeah - my inclination is to continue using the nsc CLI, if the implementation & maintenance cost of the parallel downloading is too high. Since this is caching, we assume speed is of the essence, so I would rather err on the side of speed rather than usability (it means we have to document to inform users to do restore outside their containers).

But this is based on the assumption that there is a 75% drop in speed for large artifacts. Would love to see what you find.

Previous concerns have been addressed.

buildsworth-bk-app

The previous download handling concerns remain addressed, and I found no new code concerns in the retry-limit update. The direct cache transport and shared Namespace client lifecycle are above the L1 approval ceiling; the branch also currently conflicts with main in go.mod/go.sum, and build #14096 is failing.

Want to dig deeper?

Paste this into your agent to explore the findings from this review's Buildkite build:

Download the buildsworth logs from build 21111, then answer my questions about the findings.

Install the reading-buildsworth-logs skill to run this.

Amp-Thread-ID: https://ampcode.com/threads/T-01a0313b-c877-708c-a5f4-8de217bd8da9 Co-authored-by: Kate Sy <kate.sy@buildkite.com>

Previous concerns have been addressed.

buildsworth-bk-app

The rebase resolves the previous go.mod/go.sum conflict, and the download-handling concerns remain addressed; I found no new code concerns. The direct cache transport and shared Namespace client lifecycle remain above the L1 approval ceiling, and build #14097 is currently failing.

Want to dig deeper?

Paste this into your agent to explore the findings from this review's Buildkite build:

Download the buildsworth logs from build 21125, then answer my questions about the findings.

Install the reading-buildsworth-logs skill to run this.

buildkate commented Aug 24, 2026

View reviewed changes

buildkate marked this pull request as ready for review August 24, 2026 23:52

buildkate requested review from a team as code owners August 24, 2026 23:52

chatgpt-codex-connector Bot reviewed Aug 24, 2026

View reviewed changes

Comment thread internal/cache/store/nsc.go Outdated

buildsworth-bk-app Bot requested changes Aug 24, 2026

View reviewed changes

Comment thread internal/cache/store/nsc.go Outdated

buildkate force-pushed the A-1435-namespace-storage-api-phase-1 branch from 948571a to f836ac4 Compare August 24, 2026 23:58

buildsworth-bk-app Bot requested changes Aug 25, 2026

View reviewed changes

buildkate added the internal Non-user facing, internal change. label Aug 25, 2026

buildsworth-bk-app Bot previously requested changes Aug 25, 2026

View reviewed changes

This comment was marked as resolved.

Sign in to view

buildsworth-bk-app Bot previously requested changes Aug 25, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Aug 25, 2026

View reviewed changes

Comment thread internal/cache/store/nsc.go Outdated

buildsworth-bk-app Bot previously requested changes Aug 25, 2026

View reviewed changes

buildsworth-bk-app Bot reviewed Aug 25, 2026

View reviewed changes

jamiemonserrate reviewed Aug 25, 2026

View reviewed changes

buildsworth-bk-app Bot reviewed Aug 26, 2026

View reviewed changes

ampagent and others added 15 commits August 26, 2026 01:55

Add lazy Namespace storage client

45e4676

Amp-Thread-ID: https://ampcode.com/threads/T-01a0313b-c877-708c-a5f4-8de217bd8da9 Co-authored-by: Kate Sy <kate.sy@buildkite.com>

Ignore unlicensed Namespace generated modules

81e3f36

Amp-Thread-ID: https://ampcode.com/threads/T-01a0313b-c877-708c-a5f4-8de217bd8da9 Co-authored-by: Kate Sy <kate.sy@buildkite.com>

Replace NSC CLI transfers with storage API

bbc105d

Amp-Thread-ID: https://ampcode.com/threads/T-01a0313b-c877-708c-a5f4-8de217bd8da9 Co-authored-by: Kate Sy <kate.sy@buildkite.com>

Fix Namespace storage lint errors

4568939

Amp-Thread-ID: https://ampcode.com/threads/T-01a0313b-c877-708c-a5f4-8de217bd8da9 Co-authored-by: Kate Sy <kate.sy@buildkite.com>

Document Namespace SDK version pin

9d00a2e

Amp-Thread-ID: https://ampcode.com/threads/T-01a0313b-c877-708c-a5f4-8de217bd8da9 Co-authored-by: Kate Sy <kate.sy@buildkite.com>

Wire Namespace client into cache commands

f47047f

Amp-Thread-ID: https://ampcode.com/threads/T-01a0313b-c877-708c-a5f4-8de217bd8da9 Co-authored-by: Kate Sy <kate.sy@buildkite.com>

Use NSC naming for storage client

cc7cf3f

Amp-Thread-ID: https://ampcode.com/threads/T-01a0313b-c877-708c-a5f4-8de217bd8da9 Co-authored-by: Kate Sy <kate.sy@buildkite.com>

Retry transient NSC cache downloads

54a25d7

Amp-Thread-ID: https://ampcode.com/threads/T-01a0313b-c877-708c-a5f4-8de217bd8da9 Co-authored-by: Kate Sy <kate.sy@buildkite.com>

Retry HTTP2 stream resets in NSC downloads

19a6fd9

Amp-Thread-ID: https://ampcode.com/threads/T-01a0313b-c877-708c-a5f4-8de217bd8da9 Co-authored-by: Kate Sy <kate.sy@buildkite.com>

Recognize net HTTP2 stream reset errors

a24f28a

Amp-Thread-ID: https://ampcode.com/threads/T-01a0313b-c877-708c-a5f4-8de217bd8da9 Co-authored-by: Kate Sy <kate.sy@buildkite.com>

Harden NSC cache download handling

625c6a2

Amp-Thread-ID: https://ampcode.com/threads/T-01a0313b-c877-708c-a5f4-8de217bd8da9 Co-authored-by: Kate Sy <kate.sy@buildkite.com>

Retry transient NSC download statuses

38e34f5

Amp-Thread-ID: https://ampcode.com/threads/T-01a0313b-c877-708c-a5f4-8de217bd8da9 Co-authored-by: Kate Sy <kate.sy@buildkite.com>

Treat NSC download 404 as cache miss

105c227

Amp-Thread-ID: https://ampcode.com/threads/T-01a0313b-c877-708c-a5f4-8de217bd8da9 Co-authored-by: Kate Sy <kate.sy@buildkite.com>

Limit NSC whole-file download attempts

fb8d7db

Amp-Thread-ID: https://ampcode.com/threads/T-01a0313b-c877-708c-a5f4-8de217bd8da9 Co-authored-by: Kate Sy <kate.sy@buildkite.com>

Tidy dependencies after rebase

9488a06

Amp-Thread-ID: https://ampcode.com/threads/T-01a0313b-c877-708c-a5f4-8de217bd8da9 Co-authored-by: Kate Sy <kate.sy@buildkite.com>

buildkate force-pushed the A-1435-namespace-storage-api-phase-1 branch from 9c7480c to 9488a06 Compare August 26, 2026 01:58

buildsworth-bk-app Bot reviewed Aug 26, 2026

View reviewed changes

buildkate marked this pull request as draft August 27, 2026 01:32

Uh oh!

Conversation

buildkate commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Context

Changes

Testing

Disclosures / Credits

Uh oh!

socket-security Bot commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

buildkate Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

buildkate Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

buildsworth-bk-app Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

buildsworth-bk-app Bot left a comment

Choose a reason for hiding this comment

Uh oh!

buildkate commented Aug 25, 2026

Uh oh!

buildkate commented Aug 25, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 25, 2026

Uh oh!

buildsworth-bk-app Bot left a comment

Choose a reason for hiding this comment

Uh oh!

buildkate commented Aug 25, 2026

Uh oh!

buildkate commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

buildkate commented Aug 25, 2026

Uh oh!

buildkate commented Aug 25, 2026

Uh oh!

buildkate commented Aug 25, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 25, 2026

Uh oh!

buildsworth-bk-app Bot left a comment

Choose a reason for hiding this comment

Uh oh!

buildkate commented Aug 25, 2026

Uh oh!

buildkate commented Aug 25, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

buildsworth-bk-app Bot left a comment

Choose a reason for hiding this comment

Uh oh!

buildkate commented Aug 25, 2026

Uh oh!

buildkate commented Aug 25, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 25, 2026

Uh oh!

buildsworth-bk-app Bot left a comment

buildkate commented Aug 24, 2026 •

edited

Loading

socket-security Bot commented Aug 24, 2026 •

edited

Loading

buildkate commented Aug 25, 2026 •

edited

Loading