test: expand audit output BDD coverage — fan-out resilience, CEF format, cross-output correlation

[millerjp]

Problem

The audit_outputs.feature file has 12 scenarios covering basic audit output functionality, but the multi-output fan-out and resilience testing is thin. The infrastructure is already in place (syslog-ng TLS container, webhook receiver with /shutdown//reset endpoints, Docker Compose overlay, TLS certs) — it's the scenario coverage that needs expanding.

Current coverage

Area	Scenarios	Notes
File output isolation	2	Field presence + JSON format
Webhook output isolation	3	schema_register, subject_delete_soft, config_update
Syslog TLS isolation	3	schema_register, app_name, subject_delete_soft
Multi-output fan-out	1	Single event verified across file + webhook + syslog
Multiple events to webhook	1	Checks webhook has ≥2 events (does NOT verify they also reach file/syslog)
Failure event recording	1	422 Invalid Schema to file + webhook
Graceful shutdown flush	1	File output only

What's missing

1. Output failure resilience (0 scenarios)
The webhook receiver already exposes /shutdown and /reset endpoints for simulating outages. No scenario tests what happens when an output fails:

• Server continues operating when webhook is unreachable
• Events still delivered to healthy outputs (file, syslog) when webhook is down
• Events resume to webhook after it recovers (/reset)
• Syslog connection loss doesn't crash the server
• File output continues if syslog is unreachable

2. Buffer overflow behaviour (0 scenarios)
Webhook is configured with buffer_size: 10000. No scenario tests:

• Behaviour when buffer fills (are events dropped gracefully?)
• Metrics increment on buffer drop (audit_buffer_drops_total)
• Healthy outputs still receive events when one output's buffer is full

3. Batch flushing (0 scenarios)
Webhook is configured with batch_size: 1 and flush_interval: "100ms" in tests. No scenario tests:

• Larger batch sizes (e.g., batch_size: 10) — are events batched correctly?
• Flush interval timing — do events arrive within the configured interval?
• Partial batch flush on shutdown

4. Cross-output event correlation (0 scenarios)
No test verifies that the SAME event appears identically across all outputs:

• Same request_id in file, webhook, and syslog for a single operation
• Same timestamp across outputs
• Same duration_ms across outputs
• Field-by-field comparison proving fan-out delivers identical payloads

5. CEF format (0 scenarios)
Step definitions reference _cef_fetcher and FormatCEF() is implemented in audit_format.go, but:

• No scenario enables CEF format on any output
• No scenario validates CEF severity levels (3=reads, 5=writes, 8=auth failures)
• No scenario validates CEF field mapping

6. Multi-event fan-out (0 scenarios)
The single fan-out scenario tests ONE event. No scenario verifies:

• Multiple consecutive events all reach all outputs
• Different event types (schema_register, config_update, user_create) all fan out
• Event ordering is preserved across outputs

7. Graceful shutdown across all outputs (0 scenarios)
The existing shutdown scenario only checks file output. No scenario verifies:

• Pending events are flushed to webhook before shutdown
• Pending events are flushed to syslog before shutdown
• All outputs receive the same final set of events

---

Proposed Scenarios

Part 1: Output failure resilience

@audit-outputs @resilience
Scenario: Server continues when webhook receiver is down
  Given the webhook receiver is shut down
  When I register a schema "resilience-webhook-down"
  Then the response status should be 200
  And the audit log should contain an event:
    | event_type | schema_register |
    | ... (27 fields) |
  And the syslog TLS receiver should have received a message containing "schema_register"

@audit-outputs @resilience
Scenario: Events resume to webhook after recovery
  Given the webhook receiver is shut down
  When I register a schema "resilience-pre-recovery"
  Then the response status should be 200
  When the webhook receiver is reset
  And I register a schema "resilience-post-recovery"
  Then the response status should be 200
  And the webhook receiver should have received an event with event_type "schema_register"

@audit-outputs @resilience
Scenario: Healthy outputs receive events when one output fails
  Given the webhook receiver is shut down
  When I register a schema "resilience-healthy"
  Then the response status should be 200
  And the audit log should contain an event:
    | event_type | schema_register |
    | target_id  | resilience-healthy |
    | ... (27 fields) |
  And the syslog TLS receiver should have received a message containing "schema_register"
  # Webhook should NOT have the event (it was down)

Part 2: Cross-output event correlation

@audit-outputs @correlation
Scenario: Same event appears identically across all outputs
  When I register a schema "correlation-test"
  Then the response status should be 200
  And the audit log should contain an event:
    | event_type | schema_register    |
    | target_id  | correlation-test   |
    | request_id | *                  |
    | timestamp  | *                  |
    | ... (27 fields) |
  And the webhook receiver should have received an event matching:
    | event_type | schema_register    |
    | target_id  | correlation-test   |
    | request_id | *                  |
    | timestamp  | *                  |
  And the syslog TLS receiver should have received a message containing "correlation-test"

Part 3: Multi-event fan-out

@audit-outputs @multi-output
Scenario: Multiple event types are delivered to all outputs
  When I register a schema "fanout-register"
  And I set compatibility for subject "fanout-register" to "FULL"
  And I delete subject "fanout-register"
  Then the audit log should contain an event:
    | event_type | schema_register |
    | target_id  | fanout-register |
    | ... |
  And the audit log should contain an event:
    | event_type | config_update |
    | target_id  | fanout-register |
    | ... |
  And the audit log should contain an event:
    | event_type | subject_delete_soft |
    | target_id  | fanout-register |
    | ... |
  And the webhook receiver should have at least 3 events
  And the webhook receiver should have received an event with event_type "schema_register"
  And the webhook receiver should have received an event with event_type "config_update"
  And the webhook receiver should have received an event with event_type "subject_delete_soft"
  And the syslog TLS receiver should have received a message containing "schema_register"
  And the syslog TLS receiver should have received a message containing "config_update"
  And the syslog TLS receiver should have received a message containing "subject_delete_soft"

Part 4: Graceful shutdown flush across all outputs

@audit-outputs @shutdown
Scenario: Graceful shutdown flushes pending events to all outputs
  When I register a schema "shutdown-flush"
  Then the response status should be 200
  When the registry is gracefully restarted
  Then the audit log should contain an event:
    | event_type | schema_register |
    | target_id  | shutdown-flush  |
    | ... |
  And the webhook receiver should have received an event with event_type "schema_register"
  And the syslog TLS receiver should have received a message containing "schema_register"

Part 5: CEF format validation

@audit-outputs @cef
Scenario: CEF format output with correct severity levels
  When I register a schema "cef-test"
  Then the CEF audit log should contain "CEF:0|AxonOps|SchemaRegistry"
  And the CEF audit log should contain "schema_register"
  And the CEF audit log should contain "sev=5"

Note: This requires adding a CEF-enabled output to the test config (e.g., a second file output with format_type: cef).

---

Infrastructure

All infrastructure is already in place — no new containers needed:

Component	Status	Location
syslog-ng TLS container	Exists	docker-compose.audit-outputs.yml
Webhook receiver container	Exists	docker/webhook-receiver/main.go
Webhook /shutdown endpoint	Exists	Simulates outage (returns 503)
Webhook /reset endpoint	Exists	Restores after outage
TLS certificates	Exist	tests/bdd/certs/
Audit output config	Exists	configs/config.memory-audit-outputs.yaml
Step definitions	Exist	steps/audit_output_steps.go
Syslog fetcher	Exists	bdd_test.go:makeSyslogFetcherFile()
Webhook event fetcher	Exists	GET /events endpoint
CEF format serializer	Exists	internal/auth/audit_format.go

New step definitions needed

• Given the webhook receiver is shut down → POST /shutdown to webhook receiver
• When the webhook receiver is reset → POST /reset to webhook receiver
• When the registry is gracefully restarted → docker compose restart schema-registry + wait for health

Config changes needed

• For CEF tests: add a second file output with format_type: cef and a separate path (e.g., /tmp/audit-cef.log)

---

Acceptance Criteria

• Output failure resilience: server operates when webhook is down, events still reach healthy outputs
• Recovery: events resume to webhook after /reset
• Cross-output correlation: same request_id verified across file + webhook for same event
• Multi-event fan-out: 3+ different event types verified across all 3 outputs in one scenario
• Graceful shutdown: pending events flushed to all outputs (file + webhook + syslog), not just file
• CEF format: at least one scenario validates CEF output format and severity levels
• All new scenarios use 27-field DataTable audit assertions where applicable
• Existing 12 scenarios continue to pass

[millerjp]

Review findings — gaps still present

Verified against feature/mcp:

tests/bdd/features/audit_outputs.feature has 14 scenarios covering:

• File output (2), Webhook output (3), Syslog TLS output (3)
• Multi-output fan-out (2), Graceful shutdown (1)
• Cross-output field verification (3)

Still missing (as described in the issue):

• Fan-out resilience (webhook down, other outputs continue) — 0 scenarios
• CEF format validation — 0 scenarios
• Cross-output event correlation (same request_id across outputs) — 0 scenarios
• Webhook batching and retry behaviour — 0 scenarios
• Buffer overflow behaviour — 0 scenarios
• Syslog connection loss handling — 0 scenarios

Issue remains valid and open. The infrastructure (syslog-ng TLS container, webhook receiver with /shutdown//reset endpoints) exists but the scenario coverage is thin.