About LocalClicky
Overview
LocalClicky is an open-source macOS menubar assistant that runs voice control entirely on the user's own hardware. It pairs Whisper.cpp transcription, Ollama models for reasoning and vision, PyAutoGUI for pointer control and the built-in macOS say command for speech, so no audio, screenshots or commands reach a cloud service by default. The project ships under the MIT license.
The app lives in the menubar without a Dock icon. Say "Hey Jarvis" and it opens a session, records until the user stops talking, then answers or acts. It opens and quits apps, adjusts system and Spotify volume, manages files, runs shell commands, injects JavaScript into Chrome, describes screen content, creates reminders and edits video clips with ffmpeg.
Key Benefits
- Transcribes speech and reasons locally with Whisper.cpp and Ollama, keeping voice, screenshots and commands on the machine.
- Starts hands-free on the wake word "Hey Jarvis" and chains further commands without repeating that phrase.
- Reads the display on demand and clicks targets by asking the vision model for the element's bounding box.
- Cuts, mutes, merges, speeds up and resizes video files through local ffmpeg calls with no upload step.
- Remembers the last 10 exchanges so follow-up commands keep their context inside a session.
- Falls back to the built-in macOS say voice whenever the optional Fish Audio engine is unavailable.
How It Works
Users install whisper-cpp, pull the Ollama command and vision models, add ffmpeg and the Python requirements, then grant Microphone, Screen Recording and Accessibility permissions to the Python binary in their virtual environment. During a session the microphone captures speech, voice activity detection stops the recording on silence, and Whisper turns the WAV file into text. A dismissal check ends the session on words like "bye", otherwise the command model streams a reply, calls tools such as run_shell_command or look_at_screen, and the chosen speech engine reads the answer aloud before recording restarts.
Use Cases
- Mac power users change system volume, control Spotify playback and create Desktop folders by speaking instead of hunting through menus.
- Developers run shell commands and open Chrome tabs by voice while their hands stay on other work.
- Video editors trim, mute, merge or speed up clips sitting on the Desktop without sending footage to a service.
- Users with limited mouse dexterity move the pointer and click interface elements through spoken instructions.
- People who want a quick read on visual content ask what is on the screen or in a named video file and get a summary.
- Busy users dictate reminders with natural-language dates and have them land in macOS Reminders.
Why Choose This Product
LocalClicky targets users who want voice automation without handing audio and screenshots to a vendor, and every default path stays on the local machine. The trade-off is a macOS-only build that currently runs from the terminal rather than a packaged app bundle, and the authors flag limited spatial precision in the vision model as an area for improvement.
LocalClicky Pros & Cons
- Runs transcription, reasoning and speech fully offline by default
- MIT licensed and open source
- Hands-free sessions with wake word plus voice activity detection
- Screen-aware vision model returns coordinates for automatic clicks
- Local ffmpeg video editing and clip summaries without uploads
- macOS 12+ only; Windows and Linux ports do not exist yet
- Requires terminal setup, venv and manual permission grants; no .app bundle
- Vision model has limited spatial precision, so click accuracy can suffer
- Optional Fish Audio voice sends response text to an external API
Key Features
Offline Wake Word
An openWakeWord pretrained model listens for "Hey Jarvis" on the device and needs no internet connection.
Silence Detection
Voice activity detection through webrtcvad stops recording when speech ends, with a 30-second hard cap as the fallback.
Screen-Aware Clicking
The look_at_screen tool screenshots the display, resizes it to 1280px, asks the gemma4 vision model for a bounding box and clicks its center with PyAutoGUI.
Local Video Editing
ffmpeg commands trim, mute, merge, speed up, resize and add text to clips and save the result to the Desktop.
Clip Summaries
The describe_video tool samples evenly spaced frames from a video file and summarizes what happens in it.
Mac App Control
AppleScript and zsh execution open apps, set system and Spotify volume, create folders, open Chrome tabs and run shell commands.
Reminder Creation
The create_reminder tool converts natural-language dates into AppleScript and files the reminder in macOS Reminders.
Multi-Round Tool Calling
The assistant runs up to five tool rounds, checks the results and retries before it reports back.
Switchable Voice Engines
Speech plays through macOS say by default, and setting TTS_ENGINE=fish switches to Fish Audio's S2.1-pro model with automatic fallback to say.
Session Memory
The assistant holds the last 10 exchanges in memory so follow-up commands keep context within a session.
LocalClicky Pricing
Pricing extracted from the product website and may change. Check the source for current details.
Frequently asked questions about LocalClicky
Does LocalClicky send my voice or screen to the cloud?
No. Transcription, reasoning, vision and speech run on the local machine through Whisper.cpp, Ollama and the macOS say command, so audio, screenshots and commands stay put by default. The one opt-in exception is switching TTS_ENGINE to fish, which sends the assistant's spoken response text to Fish Audio's API.
What do I need installed before running LocalClicky?
The README lists macOS 12+, Python 3.11+, roughly 8GB of free RAM, Homebrew, whisper-cpp with the base English model, Ollama with the qwen3:8b command model and gemma4:e4b vision model, ffmpeg for video editing, and the Python requirements in a virtual environment. webrtcvad-wheels is optional but recommended for silence detection.
What wake word does it listen for?
The default keyword is "hey Jarvis" from the pretrained openWakeWord model, not "Computer". Users can change WAKE_MODEL to another pretrained option such as alexa or hey_mycroft, or point WAKE_MODEL_PATH at a custom trained .onnx file, and adjust DETECTION_THRESHOLD for sensitivity.
How do I end a voice session?
Say bye, goodbye, stop listening, go to sleep or that's all, and the assistant says goodbye and returns to wake word mode. A session also expires automatically after 25 seconds of silence, a value set by SESSION_IDLE_TIMEOUT in companion.py.
How does it know where to click on the screen?
When a command needs the display, the assistant calls the look_at_screen tool, which takes a screenshot, resizes it, and sends it to the gemma4 vision model for a bounding box. The center of that box is computed and clicked automatically, and the model decides on its own when a look is needed.
Which macOS permissions does it require?
LocalClicky needs Microphone for voice recording, Screen Recording for vision screenshots, and Accessibility for cursor movement and clicks. Grant them to the python3 binary inside the virtual environment; if the file picker will not accept it, adding Terminal works because Python inherits its permissions.