Build a Private Voice Assistant with Whisper, Ollama, and Kokoro TTS

Have you ever wanted your own Jarvis? A voice assistant that listens, thinks, and speaks back - all running privately on your own hardware? Here's how to build one with Whisper.cpp, Ollama, and Kokoro TTS. No cloud, no wake-word fees, no data leaving your machine. Prerequisites Hardware: Any modern computer with a microphone Software: Python 3.10+, Ollama installed Time: ~30 minutes setup Installation 1. Install Ollama and Pull a Model ollama pull qwen3:14b 2. Install Whisper.cpp git clone https://github.com/ggerganov/whisper.cpp.git cd whisper.cpp cmake -B build && cmake --build build --config Release bash models/download-ggml-model.sh medium 3. Install Kokoro TTS pip install kokoro pyaudio requests Wiring It All Together Save this as voice_assistant.py : import subprocess import tempfile import wave import pyaudio import requests from kokoro import KPipeline OLLAMA_URL = " http://localhost:11434/api/generate " MODEL = " qwen3:14b " WHISPER_BIN = " ./whisper.cpp/build/bin/whisper-cli " WHISPER_MODEL = " ./whisper.cpp/models/ggml-medium.bin " tts_pipeline = KPipeline ( lang_code = ' a ' ) def record_audio ( duration = 5 , sample_rate = 16000 ): p = pyaudio . PyAudio () stream = p . open ( format = pyaudio . paInt16 , channels = 1 , rate = sample_rate , input = True , frames_per_buffer = 1024 ) frames = [ stream . read ( 1024 ) for _ in range ( int ( sample_rate / 1024 * duration ))] stream . close (); p . terminate () with tempfile . NamedTemporaryFile ( suffix = ' .wav ' , delete = False ) as f : wf = wave . open ( f , ' wb ' ) wf . setnchannels ( 1 ); wf . setsampwidth ( 2 ) wf . setframerate ( sample_rate ) wf . writeframes ( b '' . join ( frames )) return f . name def transcribe ( audio_file ): result = subprocess . run ([ WHISPER_BIN , ' -m ' , WHISPER_MODEL , ' -f ' , audio_file ], capture_output = True , text = True ) return result . stdout . strip () def ask_llm ( prompt ): r = requests . post ( OLLAMA_URL , json = { " model " : MODEL , " prompt " : prompt , " stream " : False }) return r . json ()[ " response " ] def speak ( text ): for result in tts_pipeline ( text ): with tempfile . NamedTemporaryFile ( suffix = ' .wav ' , delete = False ) as f : f . write ( result . audio ) subprocess . run ([ ' ffplay ' , ' -nodisp ' , ' -autoexit ' , f . name ]) # Run it print ( " Listening... " ) audio_file = record_audio ( 5 ) text = transcribe ( audio_file ) print ( f " You: { text } " ) response = ask_llm ( text ) print ( f " AI: { response } " ) speak ( response ) Run it: python voice_assistant.py Speak into your mic. Wait 5 seconds. Hear the AI respond. Performance Whisper medium on CPU: transcribes in 2-4 seconds Qwen3 14B on RTX 3060: responds in 3-5 seconds Kokoro TTS on CPU: speaks in real-time (< 1 second latency) Total round-trip: ~10 seconds on modest hardware For faster responses, use Whisper tiny or a smaller LLM like Llama 3.1 8B. Originally published on everylocalai.com