A Home Assistant integration that uses local LLMs to determine who is in which room.
Status: Alpha — Running, but expect rough edges.
Occupancy sensors tell you a room is occupied. They don't tell you who is there.
Without person-level tracking, you can't:
- Apply personal lighting preferences when someone enters a room
- Know when the whole family is gathered in one place
- Send "you've been at your desk for 3 hours" reminders to a specific person
- Distinguish between a pet and a person triggering motion
WhoLLM uses a local LLM to reason across multiple weak signals—the same way humans do:
Motion in office
+ Alice's PC is on
+ Alice's phone on home WiFi
+ Bob's car is gone
─────────────────────────────
→ Alice is in the office (high confidence)
No single sensor is definitive. Together, they're enough for a good guess.
- Multi-person + pet tracking — Track everyone in your household
- Privacy-first — All inference runs locally via Ollama
- Confidence scoring — Each prediction includes confidence level and reasoning
- Vision identification — Optional camera-based identification using vision LLMs
- Graceful degradation — Falls back through heuristics when LLM is unavailable
- Habit learning — Learns patterns from your history over time
- Flexible configuration — Map specific sensors to rooms and devices to people
- Home Assistant 2024.1+
- Ollama running locally or on your network:
curl -fsSL https://ollama.com/install.sh | sh ollama pull llama3.2 # For text reasoning (~2GB) ollama pull llava:7b # Optional: for vision identification (~4GB)
- Open HACS → Integrations → ⋮ Menu → Custom repositories
- Add
https://github.com/owens-ben/WhoLLMas an Integration - Search for "WhoLLM" and install
- Restart Home Assistant
- Go to Settings → Devices & Services → Add Integration → WhoLLM
- Download the
custom_components/whollmfolder from this repository - Copy it to your Home Assistant
config/custom_components/directory - Restart Home Assistant
- Add the integration via Settings → Devices & Services
The setup wizard guides you through:
| Option | Description | Default |
|---|---|---|
| Provider | LLM provider (Ollama recommended) | ollama |
| URL | Ollama server URL | http://localhost:11434 |
| Model | Model for reasoning | llama3.2 |
| Rooms | Rooms to track | — |
| Persons | People to track | — |
| Pets | Pets to track (optional) | — |
| Poll Interval | How often to update (seconds) | 30 |
After initial setup, configure which sensors inform each room:
- Motion sensors —
binary_sensor.office_motion - Lights —
light.office_lights - Media players —
media_player.living_room_tv - Computers —
switch.alice_pc,device_tracker.alice_pc - Doors —
binary_sensor.front_door - Cameras with AI — Cameras that detect people/animals
Associate devices with specific people:
- Alice's phone →
device_tracker.alice_iphone - Alice's PC →
switch.alice_pc - Bob's car →
device_tracker.bob_car
| Entity | Description |
|---|---|
sensor.{name}_room |
Current room for each person/pet |
sensor.vision_last_identification |
Last camera identification result |
Attributes:
confidence— Prediction confidence (0-100%)indicators— Signals that contributed to the predictionreasoning— LLM's explanation (when available)last_updated— Timestamp of last update
| Entity | Description |
|---|---|
binary_sensor.{name}_in_{room} |
Whether person/pet is in specific room |
Example: binary_sensor.alice_in_office, binary_sensor.fido_in_living_room
Trigger vision-based identification from a camera.
service: whollm.identify_person
data:
camera_entity_id: camera.living_room
detection_type: person # or "animal"Control camera auto-tracking for PTZ cameras.
service: whollm.enable_tracking
data:
camera_name: e1_zoomManually trigger vision identification for all configured cameras.
service: whollm.request_vision_updateautomation:
- alias: "Alice's office lighting"
trigger:
- platform: state
entity_id: binary_sensor.alice_in_office
to: "on"
action:
- service: light.turn_on
target:
entity_id: light.office
data:
brightness_pct: 80
color_temp_kelvin: 4000automation:
- alias: "Everyone in living room"
trigger:
- platform: template
value_template: >
{{ is_state('binary_sensor.alice_in_living_room', 'on') and
is_state('binary_sensor.bob_in_living_room', 'on') }}
action:
- service: scene.turn_on
target:
entity_id: scene.movie_modeautomation:
- alias: "Alice desk reminder"
trigger:
- platform: state
entity_id: binary_sensor.alice_in_office
to: "on"
for:
hours: 3
action:
- service: notify.alice_phone
data:
message: "You've been at your desk for 3 hours. Time for a break!"┌─────────────────────────────────────────────────────────────┐
│ Home Assistant │
├─────────────────────────────────────────────────────────────┤
│ │
│ Sensors ──┐ │
│ Lights ───┤ │
│ Media ────┼──→ Context ──→ Ollama ──→ Per-Person │
│ Devices ──┤ Builder │ Sensors │
│ Cameras ──┘ │ │
│ ▼ │
│ ┌─────────────────────┐ │
│ │ Fallback Layers │ │
│ │ 1. LLM reasoning │ │
│ │ 2. Heuristics │ │
│ │ 3. Habit patterns │ │
│ │ 4. ML models │ │
│ └─────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
The LLM is taking too long to respond. Try:
- Use a smaller model:
ollama pull llama3.2:1b - Check Ollama is running:
curl http://localhost:11434/api/tags - Increase poll interval to reduce load
Check Home Assistant logs:
Settings → System → Logs → Filter by "whollm"
Common issues:
- Ollama URL incorrect or unreachable
- Missing model (run
ollama pull <model>)
- Check the
indicatorsattribute on the sensor to see what signals are being used - Verify room-entity mappings are correct
- Consider adding more sensors to improve accuracy
- Ensure you have a vision model:
ollama pull llava:7b - Configure vision model in integration options
- Check camera entity is accessible
| Scenario | Recommendation |
|---|---|
| Slow predictions | Use smaller model (llama3.2:1b, phi3:mini) |
| High CPU usage | Increase poll interval to 60+ seconds |
| GPU available | Configure Ollama to use GPU for faster inference |
| Many people/rooms | Consider dedicated Ollama instance |
- Event-driven updates (reduce polling)
- ML model training from HA history
- More LLM providers (OpenAI, Anthropic, local llama.cpp)
- Better prompt engineering
- Dashboard card
Contributions welcome! See CONTRIBUTING.md for guidelines.
Areas that need help:
- Prompt engineering — Better prompts = better accuracy
- Testing — More test coverage
- Documentation — Examples, troubleshooting guides
- Ollama — Making local LLMs accessible
- Area Occupancy Detection — Inspiration for room-level presence
MIT License — see LICENSE for details.