Migrate on-target testing from Amiberry to Copperline - #24
Merged
Conversation
Adds the pieces for driving on-target tests through Copperline instead of Amiberry: a copperline-narrator.toml.example config (A2065 + real Roadshow stack, matching the existing HardDrives/narrator boot volume and NR0:/Narrator: repo mount), iscopperline (probes Copperline's Zorro identification board so a shared S:User-Startup can pick this boot script over the Amiberry one), and boot-net, the Copperline boot script counterpart to `boot` (Roadshow bring-up still TODO before the Say acceptance test can run against a real network stack). Ignore the local Copperline test rig directory (staged boot volume + run artifacts) the same way Amiberry's HardDrives/ is kept out of the repo. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BNgYjaNTaAhz9Xf32zEU2U
Replaces the A2065 + Roadshow plan with Copperline's [hostsocket] board in net = "host" mode -- bsdsocket.library autoboots from the board's ROM and every socket call delegates straight to a host OS socket, the same shape as Amiberry's bsdsocket_emu=true. This drops the need to install/configure a guest TCP/IP stack at all, so boot-net no longer needs the Roadshow bring-up step and is now otherwise identical to `boot`. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BNgYjaNTaAhz9Xf32zEU2U
With hostsocket "host" mode, Copperline's networking setup is identical to Amiberry's bsdsocket_emu=true -- no guest stack, no interface to bring up -- so boot-net and boot had converged to the same script bar comments. Drop boot-net and the iscopperline identification-board probe that only existed to pick between them; `boot` now boots unmodified under either emulator. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BNgYjaNTaAhz9Xf32zEU2U
Replaces the Amiberry/MCP-tool workflow description with the Copperline CLI equivalent (--benchmark-until for log-based runs, --screenshot-after for the Say acceptance test's Guru check), and notes the still-unverified claim that boots/results validated under Amiberry hold under Copperline too. Carries over the amiauth project's own Copperline gotchas (AUX: redirection, CRLF serial output, the 0.12.0/0.13.0 filesys-mount hang) as forward-looking debugging notes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BNgYjaNTaAhz9Xf32zEU2U
The host build had no automated coverage of the actual Wyoming
protocol/audio code beyond "does it compile and not segfault on bad
args" -- CI never exercised wyoming.c's framing (chunk assembly, the
data_length block Piper uses, buffer-boundary handling) or
audio_host.c's WAV writing at all.
tests/mock_wyoming_server.py is a small stdlib-only Python TCP server
that serves a fixed PCM fixture (tests/golden/fixture.pcm, a
synthetic sine wave -- not real Piper output, so this tests our
client-side code, not TTS quality) split into three unevenly-sized
chunks alternating between the inline-fields and separate
data_length-block header shapes, to exercise both paths in
wyo_read_event(). tests/run_golden.sh runs wyomingtest/saytest against
it and byte-compares their output against tests/golden/*, wired into
ci.yml's host-build job.
While generating the golden fixtures, found and fixed a real bug in
both main.c and saytest.c: the host-argv parsing loop's positional
host/port branch gated on `!g_host[0]`, but g_host is already
non-empty by then because read_config() runs unconditionally first --
so a local config/narrator.wyoming's host silently overrode a
positional argv host instead of the reverse, contrary to the code's
own comment ("On the host, argv is reliable and overrides the config
file"). Track whether argv has supplied the host yet with a separate
flag instead. run_golden.sh works around the same hazard by running
each binary from a directory with no config/narrator.wyoming to find.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BNgYjaNTaAhz9Xf32zEU2U
…tes) The first actual on-target CI coverage this repo has had -- until now, CI only cross-compiled the Amiga binaries, never ran them. Runs in ghcr.io/sidick/amiga-dev:1 (the same Bebbo m68k-amigaos-gcc 6.5.0b toolchain this project already targets, plus Copperline and its lavapipe Vulkan dependency preinstalled), so one job both cross-compiles wyomingtest and boots it under Copperline against the mock Wyoming server from the golden-file tests, reached through [hostsocket] net = "host" -> 127.0.0.1 on the runner. No licensed Kickstart needed (bundled AROS ROM). Scoped to wyomingtest deliberately: it only needs dos.library + bsdsocket.library, so this stays a minimal single-purpose boot volume rather than a full installed Workbench with AHI -- proving the real cross-compiled binary's network+Wyoming-parsing path works on target, without the bigger lift of a licensed-content-free AHI/Say environment (narrator.device/AHI/Say coverage stays manual under the developer's own Copperline or Amiberry rig for now). Reuses tests/golden/wyomingtest.pcm from the host-build golden tests as the expected output, so the same fixture now gets checked on both the host protocol code and the actual m68k binary under emulation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BNgYjaNTaAhz9Xf32zEU2U
Kickstart ROMs, a real AHI driver, a licensed Workbench install, etc. have nowhere safe to live locally without risking an accidental `git add -A`/`git add .` picking them up. nondist/* is gitignored (README.md and .gitkeep excepted, so the directory itself survives a fresh checkout) -- drop licensed content in here for local Copperline/ Amiberry testing and it can't be committed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BNgYjaNTaAhz9Xf32zEU2U
Prep work for scripted regression testing (currently these are "eyeball the log" tools): every devtest phase and failtest scenario now prints a grep-able "PASS: <name>"/"FAIL: <name>" line based on its own success criteria (io_Error/io_Actual for CMD_WRITE phases, IOERR_ABORTED for the AbortIO/CMD_FLUSH phase, a 10KB AvailMem-delta threshold for the leak audits), plus an overall PASS/FAIL summary and matching process exit code. Verified both still build clean under -Wall -Wextra -Werror via the amiga-dev cross-compiler. tests/check_audio_wav.py checks a Copperline --audio-wav capture has real (non-silent) content of at least a minimum duration -- moves Say acceptance verification beyond "no Guru" towards "audio actually played". Copperline writes WAVE_FORMAT_EXTENSIBLE with an IEEE-float SubFormat GUID rather than plain WAVE_FORMAT_IEEE_FLOAT (found by running it against a real capture -- Python's wave module doesn't support either, hence the manual RIFF parsing); verified against both synthetic silent/tone fixtures and a real 45s Copperline capture against a live Piper server (peak 0.48, correctly PASS). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BNgYjaNTaAhz9Xf32zEU2U
Prep for a devtest/failtest regression script: devtest alone opens 15+ separate connections across its phases (multiple CMD_WRITEs, 100+10 open/close cycles), which the previous one-shot-then-exit server couldn't support. New optional trailing COUNT arg (default 1, unchanged behavior) -- 0 serves until killed, for callers with no fixed connection count to wait for. Re-verified run_golden.sh and run_smoke.sh still pass unmodified (both rely on the COUNT=1 default). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BNgYjaNTaAhz9Xf32zEU2U
The biggest coverage gap flagged in review: narrator.device/ translator.library/AHI/Say -- the actual product -- had zero automated regression testing; only "eyeball a screenshot for no Guru" manual checks. tests/run_nondist.sh closes that locally (never in CI -- needs nondist/'s licensed ROM + AHI-installed Workbench): boots devtest + failtest + the Say acceptance test under Copperline in one run, greps their PASS/FAIL markers, and checks the captured --audio-wav actually contains non-silent audio via tests/check_audio_wav.py. Never modifies the tracked `boot` script. Instead it temporarily swaps nondist/HardDrives/narrator's own S/User-Startup (itself never committed, so freely editable) for one that installs the device/ library/prefs, runs devtest+failtest, then the same Say line boot uses -- restored unconditionally on exit via trap. Points everything at tests/mock_wyoming_server.py via a scratch ENV:narrator.wyoming (also cleaned up on exit) rather than the developer's own local config/narrator.wyoming, so the run is deterministic and doesn't depend on a reachable real Piper server. Building this surfaced a real mock-server bug, not a device bug: narrator.device holds one persistent connection across multiple CMD_WRITEs by design (see "Persistent connection + own task" in CLAUDE.md), same as real Piper -- but mock_wyoming_server.py only answered one request per accepted connection and then closed it, so devtest's second write on the same session came back io_Error=-105 (NWERR_PROTO, "malformed/short Wyoming response") against a socket the server had already hung up on. Fixed to loop reading/answering requests per connection until the client closes it, matching real Piper's persistent-connection behavior; re-verified run_golden.sh and run_smoke.sh still pass against the fixed server. Also documents CLAUDE.md's --audio-wav capability (previously believed uncaptureable under --benchmark-until; now verified working end to end against a real boot) and the host-side golden tests, which had never been written up. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BNgYjaNTaAhz9Xf32zEU2U
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
[hostsocket]board innet = "host"mode (the direct equivalent of Amiberry'sbsdsocket_emu=true), which letboot-net/iscopperlinebe collapsed away entirely --bootnow boots identically under either emulator.tests/run_golden.sh+tests/mock_wyoming_server.py, a deterministic canned Wyoming server) wired into CI'shost-buildjob -- this repo previously had zero automated coverage of the Wyoming protocol/audio-sink code beyond "does it compile." Found and fixed a real bug along the way: a localconfig/narrator.wyomingsilently overrode a positional host argument on the host build.copperline-smokecross-compiles and bootswyomingtestunder Copperline (bundled AROS ROM, no licensed Kickstart) insideghcr.io/sidick/amiga-dev:1, byte-comparing its output against the same golden fixture.nondist/, a gitignored local directory for licensed content (Kickstart ROMs, a real AHI driver, a licensed Workbench install) that can never be accidentally committed.devtest/failtestwith grep-ablePASS:/FAIL:markers and addstests/check_audio_wav.py(non-silence/duration check against Copperline's--audio-wavcapture).tests/run_nondist.sh: a local-only regression script (needsnondist/'s licensed content, so never runs in CI) that bootsdevtest+failtest+ the Say acceptance test under Copperline in one run and checks all of it automatically -- closing the biggest remaining gap, sincenarrator.device/translator.library/AHI/Say previously had zero automated coverage, only manual "eyeball a screenshot" checks. Building this surfaced and fixed a real mock-server bug (it only answered one request per connection;narrator.deviceholds one persistent connection across writes, same as real Piper).Test plan
All of the following were run locally against real Copperline + the real cross-compiler image, not just written:
tests/run_golden.sh-- host-side golden tests passtests/run_smoke.sh-- on-targetwyomingtestsmoke test passes (cross-compiled viaghcr.io/sidick/amiga-dev:1, run under Copperline's bundled AROS ROM)tests/run_nondist.sh-- full local regression (devtest+failtest+ Say + AHI audio-content check) passes againstnondist/'s real Kickstart + AHI-installed Workbenchdevtest/failtestrebuild clean under-Wall -Wextra -Werrorvia the cross-compilerFiled #23 for a follow-up (real
rhasspy/wyoming-piperintegration check, occasional/manual, not per-PR).Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01BNgYjaNTaAhz9Xf32zEU2U