Skip to content

fix(pi): omit unsupported GPT off-reasoning effort - #223

Open
dgokeeffe wants to merge 3 commits into
databricks:mainfrom
dgokeeffe:fix/1m-context-claude-gpt-pi
Open

fix(pi): omit unsupported GPT off-reasoning effort#223
dgokeeffe wants to merge 3 commits into
databricks:mainfrom
dgokeeffe:fix/1m-context-claude-gpt-pi

Conversation

@dgokeeffe

@dgokeeffe dgokeeffe commented Jul 18, 2026

Copy link
Copy Markdown

Issue and stack

Closes #286.

Depends on #217 and #333. Review only the concern-isolated two-file incremental range:

dgokeeffe/ucode@f299e50...94fe107

Do not merge before #333. This PR was previously described as the context-window PR; #333 now owns that concern, while this existing head is retained for the GPT thinking-off compatibility commit without rewriting history.


Unit 02 — Pi GPT reasoning compatibility

Objective and user-visible behavior

Prevent Pi's thinking-off mode from emitting unsupported reasoning.effort: "none" for GPT gateway models. Explicit thinkingLevelMap: {"off": null} makes Pi omit the reasoning field when thinking is off.

Candidate: review/pi-gpt-reasoning (94fe107), mandatory PR target review/shared-context-policy (f299e50). This is a concern-isolated two-file stacked unit and must not remain conflated with context-policy PR #223.

Exact scope

Production: src/ucode/agents/pi.py.
Tests: tests/test_agent_pi.py.
Non-goals: token/context windows (unit 03), OpenCode, OSS models, MLflow proxying, cache retention.

Before / after reproduction

git show f299e50:src/ucode/agents/pi.py | rg 'thinkingLevelMap' || echo '<absent>'
# <absent>

git show review/pi-gpt-reasoning:src/ucode/agents/pi.py | rg 'thinkingLevelMap'
# entry["thinkingLevelMap"] = {"off": None}

Focused validation:

uv run --frozen pytest tests/test_agent_pi.py -q
# 63 passed
uv run --frozen ruff check .
# All checks passed

Full suite at the stacked candidate:

uv run --frozen pytest -q
# 2 failed, 1616 passed, 36 skipped

Failures are the installed Claude and Pi User-Agent capture tests. This older stack does not contain unit 01; the integrated dev baseline passes the Pi capture test, while independently reproducing the Claude capture failure.

Live Databricks evidence

Retained prior redacted gateway evidence: GPT-5-family requests with explicit effort none returned BAD_REQUEST; omitting reasoning was accepted on /ai-gateway/codex/v1. No bearer tokens or workspace identifiers are stored. The deterministic tests cover GPT-5, mini, nano, 5.5-pro, and 5.6 IDs plus a non-GPT control.

Impact map

  • Agent/provider: Pi databricks-openai / Responses only.
  • Config: one per-model metadata field in generated models.json.
  • Compatibility: affects only models already marked reasoning; non-GPT Responses entries remain unchanged.
  • Managed config/MCP/skills: no changes.

Rollback and residual risk

Revert 94fe107; rollback risks restoring gateway 400s when users select thinking off. Residual risk: future Pi semantics for a null off mapping could change, so the generated model-entry contract should remain covered by tests.

Hygiene

git diff --check review/shared-context-policy..review/pi-gpt-reasoning passes. The PR must target review/shared-context-policy; its target-relative diff is the review contract. Two-file range; no generated files, credentials, uv.lock, .pi-subagents/, or goal.md; no merge markers or unresolved index entries.

Comment thread src/ucode/databricks.py Outdated
Comment thread src/ucode/agents/claude.py Outdated
@dgokeeffe

Copy link
Copy Markdown
Author

@rohita5l I refreshed this stacked PR: Claude capability/version policy is now shared by Pi and Claude Code, the Sonnet 4.5 Claude Code behavior is explicit in the description, and both threads have detailed replies and are resolved. Could you re-review and approve the fork CI run when convenient?

@dgokeeffe
dgokeeffe force-pushed the fix/1m-context-claude-gpt-pi branch from f2fbb83 to 636c3b6 Compare July 26, 2026 04:38
@dgokeeffe

Copy link
Copy Markdown
Author

Post-review update: shared capability policy now reflects Sonnet 4.6 at 1M/64k and Fable 5 at native 1M/128k with adaptive thinking but no [1m] suffix. GPT matching is boundary-safe/case-insensitive, excludes GPT-OSS from Responses selection, and preserves generic Responses endpoints while preferring the newest semantic GPT. PR body/comparison link updated.

@dgokeeffe

Copy link
Copy Markdown
Author

Pushed 8d66253 to fix a 400 this PR would otherwise introduce.

_pi_gpt_model_entry set reasoning: True without a thinkingLevelMap. Pi's Responses builder treats a missing off-state as effort: "none" (pi-ai/dist/api/openai-responses.js, the thinkingLevelMap?.off !== null branch):

else if (model.provider !== "github-copilot" && model.thinkingLevelMap?.off !== null) {
    params.reasoning = { effort: (model.thinkingLevelMap?.off ?? "none") };
}

"none" is only valid on gpt-5.1+, so with thinking off, four ids rejected every request:

BAD_REQUEST: Unsupported value: 'none' is not supported with the 'gpt-5' model.
Supported values are: 'minimal', 'low', 'medium', and 'high'.

Measured against /ai-gateway/codex/v1/responses:

Model effort:"none" effort:"minimal" omitted
gpt-5, gpt-5-mini, gpt-5-nano 400 200 200
gpt-5-5-pro 400 400 200
gpt-5-1/-2/-3-codex/-4/-4-mini/-4-nano/-5/-6-luna/-6-sol/-6-terra 200 200 200

Fix is entry["thinkingLevelMap"] = {"off": None}, which makes Pi omit reasoning entirely — 200 on all 14 ids. {"off": "minimal"} was the other option but gpt-5-5-pro rejects it too, so it would need a hardcoded per-model exclusion list against a dynamically discovered model set. Same {"off": None} pattern we already use for the Gemini 3.x entries.

Also bisected the rest of Pi's Responses payload against the gateway; all 200, no other compat flags needed: store:false, prompt_cache_key, prompt_cache_retention:"24h", prompt_cache_options, include:["reasoning.encrypted_content"], developer role, flat tool schemas with additionalProperties:false, and the underscore session_id / x-client-request-id affinity headers.

Added two regression tests in tests/test_agent_pi.py (every gpt-5 entry pins {"off": None}; non-gpt codex entries stay bare) plus a docstring note on why the override is mandatory whenever reasoning: True is declared.

uv run pytest --ignore=tests/test_e2e.py --ignore=tests/test_e2e_tracing.py → 1102 passed, 6 skipped. The two test_e2e_user_agent failures I see locally are pre-existing on this branch (identical at 4984d37 without my commit) and need local agent binaries, so they skip in the test job.

Two things for reviewers:

  1. _pi_claude_model_entry in this same diff has the same shapereasoning: True, no thinkingLevelMap. It doesn't bite today because Claude goes through anthropic-messages, which never reaches the Responses reasoning branch. Left alone deliberately, but it's a latent trap if that ever moves.
  2. The trade in this fix: thinking-off is now delegated to the gateway default rather than explicitly requested as zero reasoning. That seemed clearly better than 400ing, but flagging it in case exact off-semantics matter.

Worth noting this was painful to diagnose because the gateway returns {"error_code","message"} instead of OpenAI's {"error":{...}}, so Pi's error-body.js recovery no-ops and every 400 renders as OpenAI API error (400): 400 status code (no body). Reported upstream: earendil-works/pi#7748. Tracking issue: #286.

@dgokeeffe

dgokeeffe commented Aug 7, 2026

Copy link
Copy Markdown
Author

Rebased this PR stack onto current main (67429ec) and force-pushed the existing branch. New head: 94fe107.

Stack dependency

This is 2 of 3 and must land after #217:

  1. feat(pi): add validated GLM/Kimi MLflow provider #217 — Pi MLflow OSS provider
  2. fix(pi): omit unsupported GPT off-reasoning effort #223 — shared Claude/GPT metadata + Pi GPT thinking-off fix (this PR)
  3. fix(opencode): route GPT models and discover OSS fallbacks #239 — OpenCode GPT routing + OSS discovery fallback

The rebase preserves main's #282 contract: Pi pins codex_models[0] verbatim; it does not re-sort for the newest GPT. The preferred_gpt_model helper is not part of this PR anymore—it is introduced with its OpenCode consumer in #239.

94fe107 contains the thinkingLevelMap: {"off": None} fix discussed above and tracked in #286. It prevents Pi from sending unsupported reasoning: {effort: "none"} to gpt-5/-mini/-nano/-5-5-pro.

If #217 is squash-merged, please rebase this branch onto the updated main before merging.

Validation

  • CI-parity suite: 1,648 passed, 6 skipped
  • Only local failures: the two pre-existing CLI capture-server tests (TestClaudeUserAgent, TestPiUserAgent), which skip in CI's unit-test job
  • ruff check and ruff format --check: clean
  • No committed uv.lock churn or conflict markers

Managed allowlist × live-discovery intersection is deliberately outside this PR and tracked in #290.

Expose the GLM and Kimi coding-model cohort through Pi and OpenCode with shared token limits and reasoning metadata. Keep unsupported chat models out of discovery, including Inkling until gateway issue databricks#215 is fixed, and retain the GPT-OSS Responses API routing guard.
Centralize Claude family/version parsing so Pi metadata, adaptive-thinking compatibility, and Claude Code's [1m] selector cannot drift. Cover Sonnet 4.5, Opus 4.6, future major versions, Fable fallback, and prefixed model IDs.
`_pi_gpt_model_entry` declared `reasoning: True` without an off-state, so for
the thinking-off case Pi's Responses builder fell back to
`reasoning: {effort: "none"}` (pi-ai openai-responses.js, the
`thinkingLevelMap?.off !== null` branch). `"none"` is only valid on gpt-5.1+,
so every request to gpt-5, gpt-5-mini, gpt-5-nano and gpt-5-5-pro was rejected:

  BAD_REQUEST: Unsupported value: 'none' is not supported with the 'gpt-5'
  model. Supported values are: 'minimal', 'low', 'medium', and 'high'.

Setting `thinkingLevelMap: {"off": None}` makes Pi omit `reasoning` entirely,
which the gateway accepts for all 14 codex ids. Verified against
/ai-gateway/codex/v1/responses: effort="none" 400s on gpt-5/-mini/-nano/-5-5-pro
and 200s on gpt-5-1..-5-6; omitting `reasoning` is 200 everywhere.
`{"off": "minimal"}` was rejected as an alternative because gpt-5-5-pro 400s on
it too. Same pattern already used for the Gemini 3.x entries.

The rest of Pi's Responses payload was bisected against the gateway and is
fine: store:false, prompt_cache_key, prompt_cache_retention:"24h",
prompt_cache_options, include:["reasoning.encrypted_content"], developer role,
flat tool schemas, and the session_id / x-client-request-id affinity headers.

Regression was hard to spot because the gateway returns
{"error_code","message"} rather than OpenAI's {"error":...}, so Pi's
error-body.js recovery no-ops and every 400 renders as
"OpenAI API error (400): 400 status code (no body)". Reported upstream as
earendil-works/pi#7748.

Refs databricks#286
@dgokeeffe
dgokeeffe force-pushed the fix/1m-context-claude-gpt-pi branch from 15ad5c0 to 94fe107 Compare August 7, 2026 10:58
@dgokeeffe dgokeeffe changed the title fix: declare long-context Claude and GPT metadata fix(pi): omit unsupported GPT off-reasoning effort Aug 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pi: gpt-5/mini/nano/5-5-pro always 400 — reasoning.effort "none" sent because thinkingLevelMap is unset

2 participants