Voice mode and the face you give Mirror

Mirror is voice-first by default. When you tap into voice mode the agent moves through four discrete states (idle, listening, processing, speaking) with explicit visual cues for each. The lip-sync on the avatar is driven by the same audio frequencies that the TTS engine streams to your speakers, so the mouth movement actually matches the sound rather than running on a generic loop.

The four states in detail

1. Idle: the avatar breathes (subtle scale animation), eyes occasionally drift and blink (random 2–6s intervals), and a faint particle field drifts around the head. No audio capture. No network activity.
2. Listening: microphone permission has been granted and ElevenLabs Scribe Realtime is transcribing one encrypted live stream. Mirror switches the presence to its listening state. For an explicit voice message, stable transcript segments accumulate without sending; the composer shows separate cancel and finish actions, and only the check submits the complete take.
3. Processing: the transcript has been sent to Mirror and the agent is composing a response. The avatar relaxes, the particle field slows, and a thin shimmer moves around the perimeter of the head to indicate "thinking" without committing to a fake-progress bar.
4. Speaking: the TTS engine is streaming synthesised audio and the lip-sync engine is driving the mouth morph targets from the AnalyserNode's frequency bins. As soon as the audio queue empties the state returns to idle.

Speech-to-text path

Mirror uses ElevenLabs Scribe v2 Realtime with global language detection, so Spanish, English, Chinese, Japanese, and code-switched speech are not restricted to the visible UI language. An explicit voice message uses manual commit: provider “committed” segments only stabilize the draft, and the check is the sole action that sends it to Mirror. Total Voice uses voice-activity detection plus a short semantic grace window, so a natural pause does not immediately become a turn. Audio stays in the active microphone stream and is not stored in localStorage; the current implementation does not retain a replayable audio copy for a second transcription pass. If Scribe cannot initialize and the browser supports Web Speech, Mirror uses that only as a degraded fallback.

Text-to-speech and lip-sync

Mirror streams its response as text and as audio in parallel. The text appears in the transcript immediately, and the audio is synthesised by a low-latency TTS provider and played back through the Web Audio API. An AnalyserNode in the graph exposes the live frequency spectrum, which the lip-sync engine reduces to morph-target weights for the visible face (mouthOpen, jawOpen, lipPucker, lipSpread) every frame based on the audio that is currently playing.

Giving Mirror your face

On a first visit Mirror shows you a generic point-cloud face. If you want, you can capture your own face from the avatar settings panel: Mirror asks for camera access, a standard on-device face-landmark model detects facial landmarks in real time, and the captured landmark cloud becomes the geometry that drives the avatar going forward. The capture step typically takes a few seconds with the user looking straight at the camera so the front-facing landmarks are stable.

The captured face is stored in your browser's local storage. On subsequent visits the platform reads from local storage and reconstructs the avatar geometry without any network round-trip. There is no server-side copy of your face data anywhere.

Reduced motion support

When you set prefers-reduced-motion in your operating system, Mirror disables the breathing animation, the eye drift, the brow raises, and the particle field. Lip-sync stays on because it is functional rather than ornamental. The four states still display, but transitions become instant rather than animated.

Failure modes and what to do

• Microphone permission denied: voice mode shows a permission prompt with a deep link to your browser's site settings. Text mode remains fully usable.
• Camera permission denied during face capture: the capture step fails gracefully and the default face is retained.
• Browser without Web Speech: Scribe still works when the browser supports getUserMedia, WebSocket, and AudioWorklet. Web Speech is only the fallback when Scribe cannot initialize.
• Voice provider outage: Mirror falls back to browser recognition when available; otherwise it surfaces the failure and leaves the text composer usable.
• Slow connection: the platform buffers briefly before resuming playback rather than breaking up audio into stutter.