Idea
tests/mock_wyoming_server.py (used by run_golden.sh, run_smoke.sh, run_nondist.sh) is a canned, deterministic stand-in for Piper -- it encodes our own assumptions about the Wyoming wire format (line-delimited JSON header, optional data_length block, payload_length binary payload, persistent multi-request connections). It's good for fast, offline, byte-exact regression testing, but it can't catch a case where those assumptions are subtly wrong and real Piper actually behaves differently.
Complement it with an occasional integration check against the real rhasspy/wyoming-piper Docker image:
- Content-based, not byte-exact (real Piper output isn't guaranteed byte-identical run-to-run or host-to-host -- same reason we don't do byte-exact Copperline audio golden tests either).
- Reuse
tests/check_audio_wav.py's non-silence/duration approach for verification, and/or a basic wyomingtest/saytest run against it checking for a sane latency + non-empty PCM.
- Not per-PR: pulling the image + a voice model and paying the one-time model-load latency (~1.8s per CLAUDE.md) is too slow/flaky for every push. Better as a manual
workflow_dispatch job or a weekly cron, separate from ci.yml's existing jobs.
Context
Raised during a testing/CI improvement pass on the copperline-migration branch (see tests/run_golden.sh, tests/run_smoke.sh, tests/run_nondist.sh, tests/mock_wyoming_server.py, tests/check_audio_wav.py). Not implemented yet -- filed for later.
Idea
tests/mock_wyoming_server.py(used byrun_golden.sh,run_smoke.sh,run_nondist.sh) is a canned, deterministic stand-in for Piper -- it encodes our own assumptions about the Wyoming wire format (line-delimited JSON header, optionaldata_lengthblock,payload_lengthbinary payload, persistent multi-request connections). It's good for fast, offline, byte-exact regression testing, but it can't catch a case where those assumptions are subtly wrong and real Piper actually behaves differently.Complement it with an occasional integration check against the real
rhasspy/wyoming-piperDocker image:tests/check_audio_wav.py's non-silence/duration approach for verification, and/or a basicwyomingtest/saytestrun against it checking for a sane latency + non-empty PCM.workflow_dispatchjob or a weekly cron, separate fromci.yml's existing jobs.Context
Raised during a testing/CI improvement pass on the
copperline-migrationbranch (seetests/run_golden.sh,tests/run_smoke.sh,tests/run_nondist.sh,tests/mock_wyoming_server.py,tests/check_audio_wav.py). Not implemented yet -- filed for later.