Home Assistant Voice Control Recipes

GPU/CUDA-accelerated voice control stack for Home Assistant. Runs on x86/x64 and GB10 ARM64 devices (including the NVIDIA DGX Spark).

100% Local - No Cloud, No Subscriptions, No Data Leaving Your Network

Every component in this stack runs entirely on your own hardware. Your voice commands, transcriptions, conversations, and responses never leave your local network - no cloud APIs, no third-party services, no internet connection required after initial setup.

Architecture Overview

graph TD
    A("👂 <b>Wake Word Detection</b><br><i>OpenWakeWord</i><br><code>:10400</code>")
    B("🎙️ <b>Speech-to-Text</b><br><i>ONNX ASR + Voice Match</i><br><code>:10300 · :10350</code>")
    C("🧠 <b>Conversational Agent</b><br><i>Gemma4-26B-A4B · llama.cpp</i><br><code>:8080</code>")
    D("🔊 <b>Text-to-Speech</b><br><i>Kokoro FastAPI</i><br><code>:8880 · :10900</code>")

    A -- "audio stream" --> B -- "transcript" --> C -- "response text" --> D

    style A fill:#4a90d9,stroke:#2a6cb8,color:#fff
    style B fill:#6b8e23,stroke:#4a6b14,color:#fff
    style C fill:#d4772c,stroke:#b35d1a,color:#fff
    style D fill:#9b59b6,stroke:#7d3c98,color:#fff

Loading

Every component runs as a Docker container with NVIDIA GPU passthrough, communicating via the Wyoming protocol - Home Assistant's native voice satellite interface.

---

1. 👂 Wake Word Detection - OpenWakeWord

Directory: wake-word/openwakeword/

Listens for a wake word to activate the voice pipeline. Uses the okay_nabu model by default, with support for custom wake word models.

Setting	Value
Image	rhasspy/wyoming-openwakeword:latest
Port	10400 (TCP + UDP)
Wake Word	okay_nabu
Threshold	0.65
Trigger Level	3

Custom models can be placed in /opt/models/wyoming-openwakeword/custom and will be available in the /custom directory inside the container.

cd wake-word/openwakeword
docker compose up -d

---

2. 🎙️ Speech-to-Text (STT) - Wyoming ONNX ASR

Directory: speech-to-text/wyoming-onnx-asr/

Converts speech audio into text using GPU-accelerated ONNX models. The recommended model is NVIDIA NeMo Parakeet TDT 0.6B v2 - a fast, accurate ASR model optimized for streaming speech recognition.

Setting	Value
Port	10300
Model	nemo-parakeet-tdt-0.6b-v2

Alternative models (commented out in the compose file):

• onnx-community/whisper-large-v3-turbo
• mekpro/whisper-medium-turbo

x86/x64 — Uses the upstream wyoming-onnx-asr image by tboby via compose.yaml:

cd speech-to-text/wyoming-onnx-asr
docker compose up -d

ARM64 (DGX Spark) — Uses a fork that adds GB10 ARM64 support via compose.arm64.yaml:

cd speech-to-text/wyoming-onnx-asr
docker compose -f compose.arm64.yaml up -d

Voice Extraction & Speaker Verification - Wyoming Voice Match

Directory: speech-to-text/wyioming-voice-match/

A Wyoming protocol ASR proxy that extracts your voice from background noise before forwarding audio to the downstream STT service. This solves two common problems: false activations triggered by TVs or radios, and noisy transcripts contaminated with background audio.

How it works:

1. Audio buffering - Captures incoming audio after wake word detection
2. Speaker verification - Analyzes the loudest segment against enrolled voiceprints using ECAPA-TDNN neural speaker embeddings. If no match is found, the pipeline stops silently - preventing false activations
3. Speaker extraction - Divides the full audio into speech regions, keeping only regions that match your voiceprint. Background voices (TV, radio, other people) are discarded
4. ASR forwarding - Sends the cleaned audio - containing only your voice - to the downstream STT service for transcription

The result is dramatically cleaner transcriptions, especially in noisy environments.

Setting	Value
Image	ghcr.io/jxlarrea/wyoming-voice-match:latest
Port	10350
Upstream	tcp://<ASR_HOST>:10300
Verify Threshold	0.26
Extraction Threshold	0.25
Require Speaker Match	true

Voice Match sits in front of the ONNX ASR service as a proxy. Update UPSTREAM_URI to point to your ASR instance. When REQUIRE_SPEAKER_MATCH=true, only enrolled speakers can trigger commands. Speaker verification runs in 5-25ms on GPU.

cd speech-to-text/wyioming-voice-match
docker compose up -d

---

3. 🧠 Conversational Agent (LLM) - Gemma-4-26B-A4B-it-Q8

Directory: conversational-agent-llm/

The brain of the pipeline. Runs Gemma-4-26B-A4B (Q8_0 quantization) with MTP speculative decoding for significantly faster inference. Served via llama.cpp with an OpenAI-compatible API. If you don't have the VRAM to run the 26B model, Gemma-4-E4B-it-GGUF (Q8_0 quantization) is a solid alternative for GPUs with less memory.

A sample system prompt is included in system-prompt.txt. It configures the LLM as a concise voice assistant that maps natural language commands to Home Assistant scripts and tool calls. The prompt defines script mappings for common phrases (e.g. "make it cozy" triggers script.ai_master_bedroom_cozy), enforces tool argument rules, and keeps responses short and plain text for TTS output. Use it as a starting point and customize it with your own scripts and devices.

How It Works

The LLM component uses a two-layer architecture: a base image and model-specific compose stacks.

Step 1: Build the Base Image

The llama-base-image/ directory contains Dockerfiles that compile llama.cpp from source and bundle it with llama-proxy (a Go-based logging proxy) into a single Docker image tagged llama-server:latest.

Use build.sh to build the base image:

cd conversational-agent-llm/llama-base-image

# x86_64 (default)
./build.sh

# ARM64 / DGX Spark
./build.sh arm64

The x86 build uses Dockerfile which auto-detects CUDA architectures. The ARM64 build uses Dockerfile.arm64 which targets CUDA architecture 121 (Blackwell/GB10) specifically.

You only need to rebuild this image when llama.cpp itself gets updated.

Step 2: Launch a Model

Each model has its own compose stack that uses the llama-server:latest base image. Pick the model that fits your VRAM:

• llama-gemma4-26B-A4B-it-Q8/ — Gemma-4-26B-A4B-it-Q8 with speculative decoding (recommended)

cd conversational-agent-llm/llama-gemma4-26B-A4B-it-Q8
docker compose up -d

Place your GGUF model files in /opt/models/llama-server/. The compose files mount this directory as read-only at /models inside the container.

Llama-Proxy (Request Analytics)

The Gemma-4-26B-A4B compose stack includes llama-proxy, a transparent proxy that sits between Home Assistant and llama-server. It uses the same llama-server:latest base image but runs the llama-proxy binary instead.

Home Assistant (:8080) → llama-proxy → llama-server (:8081)

When configuring the LLM in Home Assistant, point it to the proxy port (http://<host>:8080/v1), not directly to llama-server. The proxy forwards all requests transparently while capturing metrics:

• Request/response latency
• Token counts and tokens per second
• Speculative decoding draft acceptance rates
• KV cache hit rates
• Tool calls and their arguments
• Full conversation history per request

All metrics are viewable in a built-in web dashboard at http://<host>:9090.

Port	Service
8080	llama-proxy (Home Assistant connects here)
8081	llama-server (direct access if needed)
9090	Dashboard (request analytics)

Key Configuration

Parameter	Value	Purpose
Main Model	gemma-4-26B-A4B-it-Q8_0.gguf	Primary inference model
Draft Model	gemma-4-E2B-it-Q4_0.gguf	Speculative decoding for faster generation
Context Window	18192 tokens	Sufficient for complex multi-turn conversations
GPU Layers	999	Offload all layers to GPU
Temperature	0.0	Deterministic output for reliable smart home control
Flash Attention	on	Faster attention computation
KV Cache Quantization	Q8_0	Reduced VRAM usage
Thinking Mode	disabled	Skips chain-of-thought for lower latency

Speculative Decoding

The draft model (gemma-4-E2B-it-Q4_0) proposes candidate tokens that the main model (gemma-4-26B-A4B-it-Q8_0.gguf) verifies in parallel. This yields significant speedups for tool-calling workloads where output patterns are predictable.

Draft Parameter	Value
--draft-max	16
--draft-min	1
--draft-p-min	0.75

---

4. 🔊 Text-to-Speech (TTS)

Directory: text-to-speech/

Converts LLM responses back to natural-sounding speech using Kokoro FastAPI, a lightweight GPU-accelerated ONNX TTS engine. A Wyoming-OpenAI bridge translates between the Wyoming protocol and the engine's OpenAI-compatible API, exposing a Wyoming endpoint on port 10900 for Home Assistant.

Kokoro FastAPI

Directory: text-to-speech/kokoro/

Kokoro FastAPI is a lightweight, GPU-accelerated ONNX TTS engine. Fast and simple — a good default choice.

Setting	Value
Port	8880
Voices	20 built-in (10 female, 10 male)
Speed	1.1x
Streaming	Enabled (min 10 words, max 220 chars)

x86/x64 — Uses the upstream image directly via compose.yaml:

cd text-to-speech/kokoro
docker compose up -d

ARM64 (DGX Spark) — The upstream GPU Dockerfile is broken on ARM64 (forces x86 base image). The included kokorofastapi-arm64-build-patch.sh script clones the repo, patches the Dockerfile, and builds a local image:

cd text-to-speech/kokoro
bash kokorofastapi-arm64-build-patch.sh

To start the ARM64 stack manually:

cd text-to-speech/kokoro
docker compose -f compose.arm64.yaml up -d

Volume Configuration: The kokoro.env file controls runtime settings like default_volume_multiplier=2.0. Place your model files in /opt/models/kokoro.

---

5. 📱 Voice Satellite

A Home Assistant custom card and integration that turns any web browser into a voice satellite - wall-mounted tablets, kiosks, or any device running the HA dashboard becomes a fully functional voice control endpoint with no dedicated hardware required.

Key features:

• In-browser wake word detection - Runs microWakeWord locally via TensorFlow Lite WebAssembly, so wake word processing happens on the device itself without hitting the server
• Continuous listening - Automatically returns to wake word mode after each conversation, behaving like a dedicated satellite
• Visual feedback - Themed activity bar showing pipeline state (listening, processing, speaking), real-time transcription display, and reactive audio visualizations. Includes built-in skins (Alexa, Google Home, Siri, Retro Terminal, and more)
• Voice-controlled timers - On-screen countdown pills with alerts
• Announcements - Receive TTS announcements via service calls with pre-announcement chimes
• Multi-turn conversations - Continue talking without repeating the wake word
• Audio processing - Built-in noise suppression, echo cancellation, auto-gain, and voice isolation
• Per-device configuration - Each browser/tablet can have its own satellite entity and settings on a shared dashboard

This is the presentation layer that ties the entire pipeline together - it captures the user's voice via the browser microphone and feeds it through the Wake Word → STT → LLM → TTS stack described above, then plays back the synthesized response.

---

Port Reference

Service	Port	Protocol
Wake Word (OpenWakeWord)	10400	TCP/UDP
Speech-to-Text (ONNX ASR)	10300	TCP
Speaker Verification (Voice Match)	10350	TCP
LLM Proxy (Home Assistant connects here)	8080	HTTP
LLM Server (llama-server direct)	8081	HTTP
LLM Dashboard	9090	HTTP
TTS Engine (Kokoro FastAPI)	8880	HTTP
TTS Bridge (Wyoming-OpenAI)	10900	TCP

---

Home Assistant Integration

Once all services are running, add them as Wyoming protocol integrations in Home Assistant:

1. Settings → Devices & Services → Add Integration → Wyoming Protocol
2. Add each service by its host and port:
   • Wake Word: <host>:10400
   • STT: <host>:10300 (or <host>:10350 if using Voice Match)
   • TTS: <host>:10900
3. Configure the Conversation Agent to use the llama-proxy instance (http://<host>:8080/v1) via an OpenAI-compatible integration
4. Create a Voice Assistant pipeline combining all four components

---

Hardware Tested

• NVIDIA DGX Spark (ARM64, GB10 GPU) - full stack, all components
• x86/x64 systems with NVIDIA GPUs (CUDA-capable)