A self-hosted, privacy-first home assistant that uses Signal as its interface. Send a text, get an answer. Send a photo, it goes to your photo frame. Ask it to turn off the lights, it does.
This started as a simple photo relay: my wife sends a photo from her phone, it automatically appears on our digital photo frame via Immich. No third-party apps, no account creation, no cloud uploads — just Signal, which we already use.
From there it grew organically. If Signal is already the interface, why not add an LLM for conversational AI? If the LLM is there, why not let it control Home Assistant devices? Each integration followed the same pattern: keep the original simple functionality, layer new capabilities on top, and make everything optional. You can run this as just a photo relay if that's all you want.
Signal Message
|
v
signal-cli-rest-api (Docker)
|
v
relay.py (message router)
|
+-- Photo? --> Immich API --> Photo Frame album
|
+-- Voice? --> Whisper STT --> [text pipeline] --> Piper TTS --> voice reply
|
+-- Text? --> Ollama (Qwen3 14B, local)
|
+--> Tool calls: Home Assistant, Immich search,
| SearXNG, Mealie recipes, Paperless docs,
| CalDAV calendar, reminders, memory
|
+--> Plain text response via Signal
Signal as the interface — No custom app to build or maintain. Works on every phone. Already end-to-end encrypted. The UX is just "send a message" — zero learning curve.
Local-first LLM — All conversational AI runs on a local Ollama instance (Qwen3 14B). No API keys needed for the core assistant, no per-message costs, no data leaving the network. The model stays loaded in VRAM for instant responses.
Everything optional — Every integration (Home Assistant, voice, web search, recipes, documents, calendar) activates only when its environment variables are set. The core is just Signal + Immich + Ollama. Start minimal, add what you need.
SQLite for state — Conversation history, reminders, persistent memory, and scheduled tasks all live in a single SQLite database. Simple, no extra services, survives restarts.
Send photos from Signal, they're uploaded to an Immich album. Supports JPEG, PNG, HEIC, WebP, GIF. Batch uploads with confirmation messages.
Natural language chat via a local LLM. Maintains per-user conversation history with automatic compaction (old messages are summarized to stay within context limits).
"Turn off the living room lights" / "What's the temperature?" / "Set the thermostat to 72" — natural language device control through Home Assistant's REST API. Entity names are injected into the system prompt so the LLM knows what devices exist.
Search your Immich photo library by description ("photos from the beach last summer") or by person (facial recognition). Photos are sent back through Signal.
Send a voice note, it's transcribed via Whisper and processed as text. Responses come back as both text and a synthesized voice note via Piper TTS.
Search the web via a local SearXNG instance. Fetch and read webpage content with RSS fallback for sites behind consent walls.
Search and retrieve recipes from Mealie. Import recipes from URLs. Add individual items or entire recipe ingredient lists to a shared grocery list.
Search scanned documents in Paperless-ngx. Find invoices, contracts, letters by keyword.
Add, list, and delete events on a shared CalDAV calendar (Radicale). Supports timed events, all-day events, and recurring schedules.
Set one-shot or recurring reminders (daily, weekly, weekdays, monthly, every N minutes). Delivered as Signal messages when due.
The assistant remembers facts about you across conversations — preferences, names, important dates. Memories are injected into every conversation automatically.
Scheduled daily briefing delivered via Signal: weather, calendar events, pending reminders, and daily notes, formatted by the LLM.
Separate module: subscribes to Frigate MQTT events for bird detections on a feeder camera, captures burst frames, uses Claude vision to select the best frame and identify species, crops and uploads to an Immich album with species tags. Correlates with BirdNET audio detections when available.
- Python 3.12 — single-process, async WebSocket or polling
- Ollama — local LLM inference (Qwen3 14B recommended, any Ollama model works)
- signal-cli-rest-api — Signal message send/receive
- Immich — photo management and CLIP-based search
- Home Assistant — device control via REST API
- Wyoming Whisper/Piper — voice transcription and synthesis
- SearXNG — private web search
- Mealie — recipe management and shopping lists
- Paperless-ngx — document management
- Radicale — lightweight CalDAV server
- SQLite — conversation history, reminders, memory, scheduled tasks
- Anthropic Claude API — vision-based bird identification (BirdSnap only)
- Docker — containerized deployment
- Docker and Docker Compose
- A registered Signal phone number via signal-cli
- An Immich instance with an API key
-
Clone the repo:
git clone https://github.com/yourusername/jarvis.git cd jarvis -
Copy and configure environment:
cp .env.example .env # Edit .env with your Signal number, Immich API key, etc. -
Build and run:
docker build -t jarvis-relay relay/ docker run --env-file .env jarvis-relay
-
Send a photo from Signal to your registered number — it should appear in your Immich album.
Each integration is enabled by setting its environment variables in .env. See .env.example for all options.
Chat: Set OLLAMA_API_URL and OLLAMA_MODEL — text messages get LLM responses.
Home Assistant: Set HA_API_URL and HA_API_TOKEN — unlocks device control tools.
Voice: Set WHISPER_API_URL and PIPER_API_URL — voice notes get transcribed and responses are spoken.
Everything else: Each section in .env.example is self-documenting.
docker build -t birdsnap birdsnap/
docker run --env-file .env birdsnapRequires Frigate for object detection and optionally BirdNET-Go for audio species ID. See the BirdSnap section in .env.example.
Create a context.txt file mounted at /app/data/context.txt to inject additional instructions into the system prompt. Useful for things like:
- Device-specific notes ("The Main Floor speaker is the Ecobee thermostat")
- Music playback instructions
- Household-specific context
relay/
relay.py # Message router (entry point)
llm.py # Ollama chat client with tool-call loop
conversation.py # SQLite conversation history with compaction
homeassistant.py # Home Assistant REST client + tools
voice.py # Whisper STT + Piper TTS via Wyoming protocol
immich_tools.py # Immich CLIP search + photo send tools
searxng.py # Web search + page fetch + RSS fallback
reminders.py # One-shot and recurring reminders
memory.py # Persistent per-user memory
heartbeat.py # Morning briefing + scheduled tasks
mealie.py # Recipe search and import
shopping.py # Grocery list management
paperless.py # Document search
caldav_tools.py # CalDAV calendar events
Dockerfile
requirements.txt
birdsnap/
birdsnap.py # Bird feeder camera + species ID
Dockerfile
requirements.txt
This is a personal project running in production daily. It works well for our household, but it's built for a specific setup. Contributions and forks welcome — the modular design makes it straightforward to add new tool modules or swap out integrations.
MIT