Proposal: Self-hosted diagnostics and executable E2E harness

[johnlanni]

Self-hosted diagnostics and executable E2E harness

Origin

This proposal follows the self-hosted server, Aone provider, evidence, webhook, runner and browser E2E dogfooding performed for #160, #161, #162 and implementation PR #163. The resulting durable behavior is archived by #165.

Why

Validating a self-hosted deployment currently requires manually composing PostgreSQL, server configuration, operator provider registries, profiles, source bindings, external code evidence, webhook receivers, browser sessions and platform-specific tests. Failures such as API/Web origin mismatch, provider-registry precedence, stale evidence, webhook suppression or an inaccessible browser route are difficult to explain without reading databases and implementation code.

What Changes

• Add profile and onboarding diagnostics that explain server identity, origins, transport posture, permissions, provider registry and planned init mutations.
• Add gate-oriented evidence explanation and synchronization diagnostics for missing, stale, wrong-revision or unauthorized evidence.
• Provide a reproducible self-hosted E2E environment lifecycle with seed data, runner, provider and browser suites and durable evidence output.
• Add webhook test delivery and policy explanation using a built-in capture fixture.
• Define platform-aware required checks and route/security matrices so local and hosted evidence can be evaluated consistently at an exact revision.

Non-Goals

• Compiling vendor-specific source or CI adapters into core.
• Shipping a hosted testing service.
• Persisting real provider credentials in fixtures or evidence bundles.
• Replacing focused unit, integration, race or browser tests.

Compatibility

Diagnostics are additive and read-only unless the user explicitly runs init, evidence synchronization or E2E seed operations. Provider-specific behavior continues through operator-registered neutral adapters.

Success Criteria

• Operators can explain profile, origin, realm, permission and provider failures before init mutates state.
• Verify failures identify the exact missing or unusable evidence fact and remediation path.
• A clean checkout can start, seed, exercise and tear down the production self-hosted stack reproducibly.
• Webhook filtering, signing, redaction, retry and replay can be tested without a custom receiver.
• Browser route, public/private authorization and platform-specific check coverage are emitted as exact-revision evidence.

[johnlanni]

Agent: Coordinator
Agent Session ID: 019f4d09-852b-75b3-a7a2-eb75adcb08fb
Agent Session Source: CODEX_THREAD_ID
Type: SPEC
ID: SPEC-001
Status: confirmed
Scope: self-hosted-diagnostics-and-e2e-harness
Links:

• Proposal Issue: N/A
• Design Issue: N/A
• Implement Issue: N/A
• Related Comments: N/A
• PR: N/A

Requirement: Profile diagnostics and init planning expose all onboarding authority

The CLI MUST provide a read-only profile doctor and init plan that explain server instance identity, canonical API and web origins, transport posture, credential binding, repository permissions, provider registry source and capability selection before any onboarding mutation occurs.

Scenario: profile doctor reports effective authority

• WHEN an operator inspects a self-hosted profile
• THEN the CLI MUST report the effective instance realm, API and web origins, trusted HTTP or TLS posture, credential source class, current identity and registry precedence without revealing secrets

Scenario: init plan identifies intended mutations

• WHEN a user plans init for an organization, repository and provider
• THEN the CLI MUST show whether it will reuse or create the server repository, source binding and workflow files and MUST report missing permissions or capabilities before mutation

Scenario: origin or realm mismatch fails safely

• WHEN the profile credential is bound to a different origin or server instance realm
• THEN doctor and init MUST fail with both expected and observed non-secret identities and MUST NOT forward the credential to the mismatched target

[johnlanni]

Agent: Coordinator
Agent Session ID: 019f4d09-852b-75b3-a7a2-eb75adcb08fb
Agent Session Source: CODEX_THREAD_ID
Type: SPEC
ID: SPEC-002
Status: confirmed
Scope: self-hosted-diagnostics-and-e2e-harness
Links:

• Proposal Issue: N/A
• Design Issue: N/A
• Implement Issue: N/A
• Related Comments: N/A
• PR: N/A

Requirement: Evidence explanation distinguishes every fail-closed gate cause

The CLI MUST explain and optionally synchronize provider-neutral evidence for a selected workflow gate and exact subject revision, distinguishing absence, staleness, revision mismatch, provider mismatch, untrusted writer, incomplete state and unavailable provider capability.

Scenario: missing evidence has actionable diagnostics

• WHEN review, verify or archive cannot consume a required evidence fact
• THEN the explanation MUST name the required fact kind, provider, subject revision, accepted states, writer policy and next supported synchronization action

Scenario: stale evidence is not silently reused

• WHEN a matching fact exists for a different revision or has been superseded
• THEN the gate MUST remain fail closed and explanation MUST identify the observed revision and current required revision

Scenario: synchronization remains provider neutral

• WHEN the selected operator provider supports evidence synchronization
• THEN the CLI MUST invoke the registered neutral capability, record immutable provenance and re-evaluate the gate without embedding vendor fields in workflow artifacts

[johnlanni]

Agent: Coordinator
Agent Session ID: 019f4d09-852b-75b3-a7a2-eb75adcb08fb
Agent Session Source: CODEX_THREAD_ID
Type: SPEC
ID: SPEC-003
Status: confirmed
Scope: self-hosted-diagnostics-and-e2e-harness
Links:

• Proposal Issue: N/A
• Design Issue: N/A
• Implement Issue: N/A
• Related Comments: N/A
• PR: N/A

Requirement: A reproducible harness exercises the production self-hosted stack

The project MUST provide an executable E2E harness that starts, seeds, exercises, records evidence for and tears down the production server, PostgreSQL, runner, neutral provider fixtures, webhook receiver and browser surfaces using isolated repeatable resources.

Scenario: environment lifecycle is isolated and resumable

• WHEN a developer starts an E2E run from a clean checkout
• THEN the harness MUST allocate deterministic isolated ports, database, secrets and fixture identities, report their non-secret handles and support safe cleanup or resume after interruption

Scenario: provider and runner flow uses exact revisions

• WHEN the suite initializes a repository and associates an external code change
• THEN the harness MUST record the source binding, provider key, external change URL, exact head revision, synchronized evidence identities and runner outcome

Scenario: evidence bundle is safe and machine readable

• WHEN the suite completes or fails
• THEN it MUST emit structured assertions, logs and artifact identities bound to the tested revision and MUST scan the bundle for credentials before retention

[johnlanni]

Agent: Coordinator
Agent Session ID: 019f4d09-852b-75b3-a7a2-eb75adcb08fb
Agent Session Source: CODEX_THREAD_ID
Type: SPEC
ID: SPEC-004
Status: confirmed
Scope: self-hosted-diagnostics-and-e2e-harness
Links:

• Proposal Issue: N/A
• Design Issue: N/A
• Implement Issue: N/A
• Related Comments: N/A
• PR: N/A

Requirement: Webhook, browser and platform policies have executable explainable coverage

The E2E and verification tooling MUST provide built-in webhook delivery tests, content-policy explanation, browser route and authorization matrices, cache-refresh assertions and platform-aware required-check evaluation at the exact implementation revision.

Scenario: webhook test proves delivery and suppression decisions

• WHEN an operator tests a subscription with human untyped, typed automation or retry fixtures
• THEN the tooling MUST verify payload compatibility, signature, destination redaction, delivery attempts, replay and the exact matched or suppressed policy reason

Scenario: browser route matrix catches navigation and authorization regressions

• WHEN the browser suite runs against seeded public and private repositories
• THEN it MUST exercise every registered sidebar and repository route, breadcrumbs, anonymous visibility, authenticated mutations and list/detail refresh consistency

Scenario: platform policy distinguishes failure from non-applicability

• WHEN a required check applies only to selected operating systems or consumes hosted CI evidence
• THEN verification MUST distinguish required failure, platform-not-applicable, accepted baseline, stale evidence and missing evidence instead of relying on free-form VERIFY prose

[johnlanni]

Issue Body Update Summary

• Issue: Proposal: Self-hosted diagnostics and executable E2E harness #168
• Target: Proposal: Self-hosted diagnostics and executable E2E harness #168

Link the diagnostics and E2E proposal to the originating self-hosted acceptance run.