eval CLI persists resolved secrets (default_headers) into the run's config.toml and log

[faresobeid]

The eval CLI dumps its resolved config into the run directory's config.toml and echoes it into eval.log. The API key is dumped by reference (api_key_var = "PRIME_API_KEY") — good — but header values are dumped resolved: a client configured with default_headers = {"X-Prime-Team-ID": <team id>} gets the actual team id persisted into both files. Anyone committing run directories as evidence/provenance ships the header value.

Downstream we carry a workaround (prime-tasks-private general-agent/src/general_agent_synth/scripts/run_eval_private.py): an mmap-based in-place redactor polling config.toml + eval.log for the secret strings while the child runs, masking them as <redacted>***. It works (masked files verified in committed batch evidence) but it is ~70 lines of race-prone machinery chasing what the dump could simply not write.

Ask: redact-at-source — either dump header values by reference like api_key_var (e.g. header_vars), or mask any resolved credential-bearing fields in the config dump and log echo. We'll delete the downstream redactor once the dump is clean.

Context: found during a review of the general-agent factory move (the redactor was flagged as over-defensive until frozen evidence proved the persistence is real).

[shriyansnaik]

had a look at this since it seemed self contained, but ran into a couple of things that make the redaction shape less obvious than it looks, so flagging before writing any code.

the main one is that masking in place fights with re-runnability. output.py documents config.toml as "the run's resolved config in the format the CLI reads (@ config.toml), so a run is re-runnable from its own output", and eval/resume.py rebuilds EvalConfig straight from that file. so a dumped headers = {"X-Prime-Team-ID": "<redacted>"} still parses and still re-runs, it just sends a junk header. that swaps a visible leak for a silent one, which feels worse.

there might be a third option beyond the two in the issue: drop the header keys the validator injected instead of masking them. apply_prime_config re-derives X-Prime-Team-ID from PRIME_TEAM_ID or the prime CLI config on every load, so leaving it out of the dump round-trips fine. downside is it needs some provenance tracking, since model_fields_set can't tell "user set headers" apart from "validator added a key to the user's headers", and it only covers the auto-injected case.

which may be the wrong case to optimise for anyway. the team id is probably not the real exposure, a user supplied default_headers carrying a bearer token for a non-prime endpoint is. covering that needs the header_vars style indirection, which is a schema change on a public config model (precedence against headers, resolution in clients/base.py, back-compat for existing config.toml files, and gepa/reflection.py copies headers across too).

also worth noting write_config is shared by eval, gepa, validate, replay and debug, so whatever lands covers all of them, and the log echo at eval/runner.py:26 is a second write site.

happy to put up a PR for whichever direction you'd prefer, just didn't want to guess between them.