Gemini Live Multimodal Engine

Ultra-Low Latency Full-Duplex AI Voice & Vision Core

Full-duplex WebSocket multimodal streaming framework for Google Gemini Live API with bi-directional PCM audio, video frames, and real-time tool calling.

Audio Latency < 120ms roundtrip

Sample Rate 24kHz PCM 16-bit

Streaming Protocol WebSocket Binary Framing

Frame Rate 15 FPS Vision Ingestion

GitHub Repository (philgear/gemini-live-multimodal-engine)

#Google Gemini Live #WebSockets #TypeScript #AudioWorklet #ADK #PCM 24kHz

Architecture Highlights

• AudioWorklet processor for zero-copy client-side mic streaming.
• WebSocket protocol handler managing Gemini Live BidiStream sessions.
• Dynamic function declaration registry for client-side tool execution.
• Automatic reconnection and conversational state snapshotting.

What is Gemini Live Multimodal Engine?

A production-grade, open-source engine designed to interact with the Gemini Multimodal Live API. It bridges raw hardware inputs (microphones, webcams, canvas renders) directly to Gemini’s bidirectional live audio-video endpoint with minimal latency.

Architecture Highlights

AudioWorklet Zero-Copy Pipeline: Captures 16kHz/24kHz raw PCM mono audio on a dedicated audio thread, avoiding main-thread UI stutter.
Bi-directional Tool Calling: Enables the AI model to pause its audio response mid-stream, execute local browser tools (e.g. database search, map lookup, device control), and resume talking seamlessly.
Resilient Network Framing: Handles packet jitter, network reconnection, and audio queue drain buffers for fluid, interruption-resistant voice chats.

// Initializing Gemini Live Multimodal Session
const liveClient = new GeminiLiveSession({
  model: 'gemini-2.0-flash-exp',
  config: {
    generationConfig: {
      responseModalities: ['audio'],
      speechConfig: {
        voiceConfig: { prebuiltVoiceConfig: { voiceName: 'Puck' } }
      }
    }
  }
});

await liveClient.connect();