The open-source iOS app connecting Meta Ray-Ban smart glasses to AI assistants — cloud or fully on-device.
Your glasses. Your AI. Your rules.
"Ok Vision, what am I looking at?"
| Use Case | Example |
|---|---|
| Instant Translation | Point at a menu or sign abroad and get real-time translation |
| Hands-Free Cooking | "What's the next step?" while your hands are covered in dough |
| Smart Shopping | "Is this a good deal?" - get price comparisons and reviews |
| Travel Guide | "Tell me about this building" - instant history and facts |
| Accessibility | Describe surroundings, read text aloud, identify objects |
| Meeting Assistant | "Remember this person is John from Marketing" |
| Learning | "What plant is this?" - identify flora, fauna, landmarks |
| DIY Helper | "How do I fix this?" - show the problem, get step-by-step help |
| Fitness Coach | "Is my form correct?" - real-time posture feedback |
With OpenClaw's 56+ tools: Send emails, control smart home, set reminders, search the web - all hands-free through your glasses.
- Local (MLX): A choice of on-device models via Apple MLX — Qwen 2.5 (0.5B/3B), Gemma 2 2B, Gemma 4 E2B, SmolVLM2 2.2B — so you can trade capability for memory/speed. Private, offline, zero API cost.
- Apple Intelligence: Apple's on-device Foundation Model (iOS 26+). No download, no memory pressure (OS-managed), private and offline. Uses guided generation + Apple's native tool-calling.
- OpenClaw: Wake word activation, 56+ tools, task execution via WebSocket
- Gemini Live: Real-time voice + vision with native audio streaming
- OpenAI: GPT-4o text + vision over the Chat Completions API — works with any OpenAI-compatible endpoint (OpenRouter, Groq, local servers, etc.)
- A natural, offline, private voice (Kokoro-82M) running on-device via MLX — selectable from a Speech Engine dropdown with a voice picker.
- Apple's system voice stays the default (with Premium/Enhanced voice support); Kokoro is the upgrade when you want lifelike speech with nothing leaving the phone.
- The models search the web whenever they're unsure or asked about current things — news, weather, prices, scores — and never answer "I can't access real-time data" without trying.
- Tavily (free tier) returns real live content for the model to summarize; DuckDuckGo is the keyless fallback.
- Smart flow: local models reformulate + retry a weak query; the OpenAI backend runs a real function-calling loop (call
web_search→ refine → answer).
- Multi-turn context on the on-device and OpenAI backends: "What's the capital of France?" → "What's its population?" just works.
- Bounded per session so local memory stays safe; Apple keeps context via a reused native session.
- Teach it faces hands-free: "Ok Vision, remember this person as Sara"
- Recognize them later: "Ok Vision, who is this?"
- Runs entirely on-device (Apple Vision
computeDistance) — no cloud, no photos leave your phone - Intent is parsed by the on-device model (agentic) — any phrasing works, and it only triggers for a person actually in view
- Reliable wake word activation ("Ok Vision") for privacy — primed recognition + self-restart so it keeps listening (survives idle, replies, and glasses off/on)
- Barge-in support - interrupt AI anytime by saying "Ok Vision"
- Conversation mode - follow-up questions without wake word
- "Ok Vision stop" - stop AI mid-speech
- Audio routes correctly whether you're using the glasses or the phone alone (loud speaker, not the earpiece)
- Pick a local model, download it on demand, and delete it to reclaim storage anytime from Settings — swap between a tiny 0.5B model and a larger one as you like.
- Photo capture on voice command ("take a photo")
- Live video streaming to Gemini (1fps)
- Seamless glasses registration via Meta AI app
- Auto-reconnect with exponential backoff (12 attempts)
- Network monitoring (auto-pause on WiFi drop)
- App lifecycle handling (suspend/resume connections)
- Secure credential storage
- All API keys configurable in-app
- No code changes needed to use
- Example config files included
| Screen | Description |
|---|---|
| Voice Interface | Main conversation screen with wake word prompt, waveform visualizer, and quick actions (camera, settings) |
| Settings | Configure AI backend, glasses, voice control, and advanced options |
| AI Backend | Choose Local Gemma, Apple Intelligence (on-device), OpenClaw (tools), Gemini Live (low latency), or OpenAI |
| Glasses | Register glasses with Meta AI, view device status, control camera streaming |
- macOS with Xcode 15+
- Physical iOS 18+ device (simulator doesn't support Bluetooth; on-device MLX models need iOS 18)
- Meta Ray-Ban smart glasses
- Meta Developer account for glasses registration
- An AI backend — one of:
- Local (MLX) — no account/key needed; a choice of on-device models (Qwen 2.5, Gemma, SmolVLM). Runs on a recent iPhone (e.g. 15 Pro/16/17)
- Apple Intelligence — no key or download; needs iOS 26+ on an Apple-Intelligence device (iPhone 15 Pro and newer)
- OpenClaw instance
- Gemini API key
- OpenAI API key (or any OpenAI-compatible endpoint)
git clone https://github.com/rayl15/OpenVision.git
cd OpenVision/meta-vision
# Copy config templates
cp Config.xcconfig.example Config.xcconfig
cp OpenVision/Config/Config.swift.example OpenVision/Config/Config.swift- Go to Meta Developer Console
- Create an app or use existing one
- Enable "Wearables" capability
- Copy your App ID and Client Token
# Your Apple Team ID (from Xcode or Apple Developer Portal)
DEVELOPMENT_TEAM = ABC123XYZ
# Your app's bundle identifier
PRODUCT_BUNDLE_IDENTIFIER = com.yourname.openvision
# Meta App ID from developer console
META_APP_ID = 1234567890
# Client Token - MUST be in this format: AR|APP_ID|TOKEN
CLIENT_TOKEN = AR|1234567890|abcdef123456789
# URL scheme for Meta AI callback
APP_LINK_URL_SCHEME = openvisionopen OpenVision.xcodeproj- Select your iOS device (not simulator)
- Build and run (⌘R)
- On first launch, go to Settings → Glasses → Register
- This opens Meta AI app to grant access
- Return to OpenVision
For Gemini Live:
- Get API key from AI Studio
- Settings → AI Backend → Gemini Settings
- Paste your API key
For OpenClaw:
- Install OpenClaw
- Settings → AI Backend → OpenClaw Settings
- Enter gateway URL and auth token
You: "Ok Vision" → Wake word activates listening
You: "What's the weather today?" → AI processes and responds via TTS
You: "Take a photo" → Captures from glasses, analyzes
You: "Ok Vision stop" → Interrupts AI mid-speech
[Silence for 30s] → Conversation ends
You: "Ok Vision, start video streaming" → Switches to Gemini Live
[Camera streams at 1fps to Gemini]
You: "What am I looking at?" → Gemini sees and responds
You: "Stop video" → Returns to OpenClaw mode
| Command | Action |
|---|---|
| "Ok Vision" | Activate listening (wake word) |
| "Ok Vision stop" | Stop AI while speaking |
| "Take a photo" | Capture and analyze view |
| "What do you see?" | Describe current view |
| "Remember this person as Sara" | Enroll a face (on-device) |
| "Who is this?" | Identify the person in view (on-device) |
| "Forget Sara" / "Who do you know?" | Remove / list known faces |
| "What's today's news?" / "Weather in Tokyo?" | Web search (on-device backends) |
| "What's its population?" (as a follow-up) | Uses conversation memory |
| "Start video streaming" | Switch to Gemini Live mode |
| "Stop video" | Exit Gemini Live mode |
Commands are routed by the on-device model, so you don't need exact wording — natural phrasing works, and it searches the web on its own when it doesn't know.
| Backend | Voice | Vision | Cost / Privacy | Best For |
|---|---|---|---|---|
| Local (MLX) | Wake word + Apple STT | via a cloud backend | Free · fully on-device | Private chat, face commands, offline — pick Qwen/Gemma/SmolVLM |
| Apple Intelligence | Wake word + Apple STT | via a cloud backend | Free · on-device, no download | Private chat on iOS 26+ devices, lowest setup |
| OpenClaw | Wake word + Apple STT | Photo on request | Self-hosted | Tasks, 56+ tools, control |
| Gemini Live | Native VAD (always on) | Continuous 1fps video | Cloud API | Natural, low-latency conversation |
| OpenAI | Wake word + Apple STT | Photo on request (GPT-4o) | Cloud API · OpenAI-compatible | Cloud text + vision, cross-checking |
Face recognition, web search, and conversation memory all run on the on-device backends (Gemma, Apple) — private, no photos or queries leave your phone unless you pick a cloud backend.
| Setting | Description |
|---|---|
| AI Backend | Choose Local (MLX), Apple Intelligence, OpenClaw, Gemini Live, or OpenAI |
| Local (MLX) | Pick a model (Qwen 2.5, Gemma, SmolVLM), download it, or delete to reclaim storage |
| Apple Intelligence | On-device model status (no key or download needed; iOS 26+) |
| Web Search | Tavily key for real live results (news/prices/scores); DuckDuckGo fallback |
| OpenClaw Gateway | WebSocket URL (e.g., wss://localhost:18789) |
| OpenClaw Token | Authentication token |
| Gemini API Key | Google API key |
| OpenAI | API key, model (default gpt-4o-mini), and base URL (OpenAI-compatible) |
| Custom Instructions | Additional system prompt |
| Memories | Key-value context for AI |
| Setting | Description |
|---|---|
| Wake Word | Activation phrase (default: "Ok Vision") |
| Wake Word Enabled | Toggle wake word requirement |
| Activation Sound | Play chime on wake word |
| Conversation Timeout | Auto-end after silence (15s-2min) |
| Setting | Description |
|---|---|
| Glasses Registration | Register/unregister with Meta AI |
| Connection Status | View connected devices |
| Camera Controls | Manual stream start/stop |
┌─────────────────────────────────────────────────────────────────┐
│ OpenVision App │
├─────────────────────────────────────────────────────────────────┤
│ Views (SwiftUI) │
│ ├── VoiceAgentView Main conversation interface │
│ ├── SettingsView Configuration panels │
│ └── HistoryView Past conversations │
├─────────────────────────────────────────────────────────────────┤
│ Services │
│ ├── OpenClawService WebSocket client, auto-reconnect │
│ ├── GeminiLiveService Native audio/video WebSocket │
│ ├── OpenAIService Chat Completions + web_search loop │
│ ├── GemmaLocalService On-device MLX models (LLM + VLM) │
│ ├── AppleFoundationService Apple Intelligence (iOS 26 model) │
│ ├── LocalAgent Shared agentic routing + conversation │
│ ├── WebSearchService Web search (Tavily + DuckDuckGo) │
│ ├── KokoroTTSService On-device neural voice (Kokoro/MLX) │
│ ├── FaceRecognitionService On-device faces (Apple Vision) │
│ ├── VoiceCommandService Wake word detection, Apple STT │
│ ├── TTSService Apple text-to-speech │
│ ├── AudioCaptureService Microphone input for Gemini │
│ └── AudioPlaybackService Speaker output for Gemini │
├─────────────────────────────────────────────────────────────────┤
│ Managers │
│ ├── GlassesManager Meta DAT SDK wrapper │
│ ├── SettingsManager JSON persistence with debounce │
│ └── ConversationManager Chat history storage │
├─────────────────────────────────────────────────────────────────┤
│ External │
│ ├── Meta DAT SDK Glasses camera & registration │
│ ├── Apple MLX On-device Gemma 4 inference │
│ ├── Apple Vision On-device face recognition │
│ ├── Apple Speech Speech recognition │
│ └── AVFoundation Audio capture & playback │
└─────────────────────────────────────────────────────────────────┘
- Ensure Meta AI app is installed and you're signed in
- Enable Developer Mode in Meta AI app settings
- Check that your Meta App ID matches the developer console
- Verify
CLIENT_TOKENformat:AR|APP_ID|TOKEN - Check all Config.xcconfig values are filled in
- Ensure bundle ID matches what's in Meta Developer Console
- Check Bluetooth connection in iOS Settings
- Ensure glasses are set as audio output device
- Try disconnecting and reconnecting glasses
- Verify API key is correct
- Check internet connection
- Ensure you have Gemini API access (not all regions supported)
- App auto-reconnects up to 12 times with exponential backoff
- Check if OpenClaw server is running
- Verify gateway URL uses
wss://(notws://) for secure connection
OpenVision/
├── App/ App entry point, URL handling
├── Config/ Configuration files
├── Models/ Data models (Settings, Conversation)
├── Services/
│ ├── AIBackend/ Connection state, errors
│ ├── OpenClaw/ WebSocket client
│ ├── GeminiLive/ Native audio WebSocket
│ ├── Voice/ Wake word, STT
│ ├── Audio/ Capture & playback
│ └── TTS/ Text-to-speech
├── Managers/ Singletons (Settings, Glasses)
├── Views/
│ ├── VoiceAgent/ Main UI
│ ├── Settings/ Config screens
│ ├── History/ Chat history
│ └── Components/ Reusable UI
└── Utilities/ Extensions, helpers
- @MainActor - All managers and services are main-actor isolated
- Callbacks - Services use callbacks (not Combine) for events
- Singleton managers - GlassesManager, SettingsManager, etc.
- Exponential backoff - OpenClaw reconnects with jittered delay
# Build for device
xcodebuild -scheme OpenVision -destination 'platform=iOS,name=iPhone' build
# Install on connected device
xcrun devicectl device install app --device <DEVICE_ID> \
~/Library/Developer/Xcode/DerivedData/.../OpenVision.appContributions are welcome!
- Fork the repository
- Create your feature branch (
git checkout -b feature/amazing-feature) - Commit your changes (
git commit -m 'Add amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
- Follow Swift API Design Guidelines
- Use
@MainActorfor UI-related code - Add documentation comments for public APIs
- Keep services focused and single-responsibility
MIT License - see LICENSE for details.
- Meta Wearables DAT SDK - Glasses integration
- Apple MLX - On-device model inference (Qwen, Gemma, SmolVLM)
- Kokoro TTS - On-device neural voice (Kokoro-82M via MLX)
- Apple Foundation Models - On-device Apple Intelligence
- Google Gemini - Live audio/video AI
- OpenClaw - AI assistant framework
- OpenAI - GPT-4o text + vision
- Tavily - Live web search built for AI assistants
- DuckDuckGo - Keyless web-search fallback
Built with Swift and ❤️