HomeAssist and Tool-Calling Mini

I wanted an assistant that runs all the time, answers fast enough to feel like a conversation, lets me cut it off mid-sentence, and acts on my own calendar, music, lights and messages. I also wanted it to speak first when something matters, without waiting to be asked. In the README I called it "the first steps on the road to building a real-life Jarvis." HomeAssist is that project. Tool-Calling Mini is a companion repo where I fine-tuned a small open model to handle the assistant's tool calls on my own hardware.

From a Pi prototype to V3

The first version ran on a Raspberry Pi. You spoke, paused, and then waited five seconds of silence before your message was sent, and the reply came back as text.

V3, the version in the public repo, runs as a background service on a Mac. You say the wake word ("Hey Honey"), talk, and either say "send it" or let auto-send fire after a pause. The answer comes back as speech. You can interrupt it by talking over it, and saying "over out" ends the session instantly, even mid-reply. Most of the work between those two points went into latency and into keeping audio components from stepping on each other.

Each stage of the pipeline sits behind an interface and is chosen in one config.py through a factory, so swapping a provider is a config change:

The tools live in a separate MCP server: Kasa lights, Spotify, Google Calendar, weather, web search, texting through macOS Messages, email and news digests, the clipboard, sticky notes, and a helper that hands coding prompts to Cursor. Memory comes in three layers: context within a session, persistent facts it has learned about me, and vector search over past conversations stored in Supabase.

I trained the custom wake phrases ("Hey Honey", "over out" and a few others) myself with openWakeWord's training pipeline: Piper generates spoken samples of the phrase, AudioSet and music clips serve as negatives, and room impulse responses are mixed in so the model holds up in a real room. "Over out" is its own small model, so the assistant can detect it while it is still talking.

Splitting the voice model from the tools

The biggest change in V3 came from noticing that the realtime model was doing two jobs at once. It had to keep a natural voice turn going and also build exact tool arguments, sometimes over several rounds, and tool work blocked the voice interface while it ran.

In February 2026 I added what I called tool signal mode. The realtime model now emits only a short TOOL signal when a request needs an action. A tool subagent takes over, builds the parameters, runs the MCP tools, and passes results to a composer that writes the spoken answer. A small handoff object carries the voice's characteristics across, so the reply still sounds like the same assistant. The realtime model goes back to what it is good at, which is fast, continuous speech.

That split exposed the next bottleneck: the subagent itself. I had started with a reasoning model (o4-mini) for tool orchestration, and its reasoning overhead bought nothing there. Choosing a light switch or a calendar query does not need deliberation. I moved the subagent to gpt-4o-mini, turned off the iterative "do I need another tool?" check by default, and let one-shot tools such as sticky notes, search and calendar skip the loop entirely. I also capped calendar access at one call per request, which stopped duplicate events. Multi-step chaining is still there behind a config flag. Per my commit notes, average tool-execution latency fell from about 12.7s to 6s.

The cost is generality. The default path treats each request as a single-tool action, so a request that needs several tools in sequence only works if chaining is turned back on in config.

Keeping audio from fighting itself

A voice assistant has several components that all want the microphone or the speaker, and they conflict if two of them grab audio at once. V3 coordinates them through an explicit state machine with a transition table that rejects illegal moves:

IDLE                -> WAKE_WORD_LISTENING
WAKE_WORD_LISTENING -> TRANSCRIBING | PROCESSING_RESPONSE | SYNTHESIZING | IDLE
TRANSCRIBING        -> PROCESSING_RESPONSE | IDLE
PROCESSING_RESPONSE -> SYNTHESIZING | IDLE
SYNTHESIZING        -> IDLE | WAKE_WORD_LISTENING | TRANSCRIBING
any state           -> ERROR -> IDLE

Transitions are logged with their timings, and alongside the latency work I added per-phase timing to the tool path (MCP connect, tool execution, parsing, each orchestration step).

Wake-word and "over out" detection run in their own subprocesses. The ONNX models could end up in a bad state when an audio stream was interrupted, and a crash there should not take the whole assistant down. By default the subprocess stays warm and only pauses its stream, which resumes in about 200ms instead of the 2-3s a cold restart takes.

Instead of each component opening and closing its own input stream, a shared audio bus keeps one stream open for the whole conversation and feeds subscribers in priority order. The barge-in detector gets audio before transcription does, so interruptions register quickly, and there is no device-acquisition gap when the assistant stops talking and starts listening. That cut the switch from speaking to listening from about 300ms to about 10ms.

The shared audio bus also keeps the last three seconds of audio in a ring buffer. When you interrupt, the captured audio prefills the next transcription, so the first words are not lost. If you interrupt within three seconds of the reply starting, the new utterance is merged into your previous message. That makes corrections like "no wait, I meant tomorrow" behave the way you would expect.

Speaking first

HomeAssist has a proactive layer built on a scheduled GitHub Actions workflow. It summarizes email and news into notifications the assistant can read out on request, and it writes calendar reminders and weather alerts into Supabase as briefings, often with the opening line already written. When I say the wake word, the assistant can start speaking a pending briefing straight away, with no model call in the critical path. That is the WAKE_WORD_LISTENING -> SYNTHESIZING transition in the table above. Only some wake words trigger briefings.

The trade is more moving parts: background jobs, a storage table, and logic to mark a briefing as delivered so it is not repeated.

Tool-Calling Mini

Once tool routing was its own step, it was a natural place to try a small local model. Every tool decision was a paid API call, and the action space was fixed at 12 tools. In March 2026 I made tool routing a provider abstraction with two implementations, OpenAI and my own model, so I could switch between them without touching the orchestrator.

The model is Qwen3-4B-Instruct, fine-tuned with QLoRA through Unsloth on about 498 tool-calling examples. The base weights load in 4-bit NormalFloat, and LoRA adapters (rank 24, rank-stabilized) target every attention and MLP projection. Loss is computed only on the assistant's completions. Each example teaches the model to emit an optional short reasoning block and then a JSON array of calls, or an empty array when no tool applies:

{"role": "assistant", "content": "<think>\nWeather request for a specific date.\n</think>\n[{\"name\": \"weather\", \"arguments\": {\"specific_date\": \"tomorrow\"}}]"}

The 12 tool schemas alone take about 2,661 tokens of the 4,096-token context, which is why the client trims conversation history to fit. The pipeline runs in stages (prepare, train, export, validate) from a single config.yaml. Export merges the adapter and quantizes to GGUF Q4_K_M, and validation runs a 19-case suite covering single-tool, multi-tool and no-tool requests. The training data pipeline on the HomeAssist side generates and augments examples and can pull real utterances from the assistant's conversation store in Supabase.

Serving is a FastAPI gateway in front of llama.cpp that exposes an OpenAI-compatible /v1/chat/completions endpoint returning structured tool_calls. Clients authenticate with short-lived keys derived by HMAC-SHA256 from a shared refresh token, rotating every five minutes. The repo also has a terminal client, a web client with a workflow for approving good responses as new training data, and a test suite that scores routing, argument accuracy and output format separately. The server later gained a model registry, so a larger Qwen3-8B profile can be selected in place of the original 4B.

The limit of this approach is that the model learns the schemas it was trained on. When a tool's arguments change, it keeps emitting the old shape until I retrain it. A frontier API picks up new schemas from the prompt with no retraining, so the local model only pays off when the tool set is stable.

Where it stands

The last commit to V3 landed in July 2026. The todo system I built inside HomeAssist grew into its own public app, task-bar-to-do, a macOS menu bar todo list with an MCP server. Both HomeAssist and Tool-Calling Mini are public, and the HomeAssist repo documents the Tool-Calling Mini API so the two can be run together.