Realtime Providers

ADK-Rust speaks two realtime backends behind the same RealtimeModel interface, so an application can offer both and switch per session. This page covers the models, voices, endpoints, auth, and how to choose.

OpenAI Realtime (GA)

use adk_realtime::openai::OpenAIRealtimeModel;
use adk_realtime::model::BoxedModel;
use std::sync::Arc;

let model: BoxedModel = Arc::new(OpenAIRealtimeModel::new(
    std::env::var("OPENAI_API_KEY")?,
    "gpt-realtime",        // or "gpt-realtime-2" (reasoning)
));

• Transport: WebSocket (openai feature) or WebRTC (openai-webrtc, needs cmake).
• Audio: 24 kHz PCM16 in and out.
• Voices: marin (a natural GA voice), alloy, and others; set with RealtimeConfig::with_voice("marin").
• Auth: OPENAI_API_KEY. The key lives on your server (never the browser).

Gemini Live

use adk_realtime::gemini::{GeminiLiveBackend, GeminiRealtimeModel};

let model: BoxedModel = Arc::new(GeminiRealtimeModel::new(
    GeminiLiveBackend::studio(std::env::var("GEMINI_API_KEY")?),
    "models/gemini-3.1-flash-live-preview",
));

• Transport: WebSocket (gemini feature, AI Studio) or Vertex AI Live (vertex-live, OAuth2 / ADC — see Realtime Agents).
• Audio: 16 kHz PCM16 in, 24 kHz PCM16 out.
• Voices: Kore and others; with_voice("Kore").
• Auth: GEMINI_API_KEY (or GOOGLE_API_KEY).

Model names differ by endpoint. AI Studio (API-key) uses names like models/gemini-3.1-flash-live-preview; Vertex/Agent Platform uses different names. The crate's GeminiLiveBackend::studio(...) targets AI Studio over the v1alpha endpoint (which is also what affective dialogue requires).

Choosing a model

• General voice + tools → gpt-realtime or gemini-3.1-flash-live-preview. Both call tools reliably. Gemini is the better fit for continuous video.
• Most natural voice / emotion-aware → Gemini native-audio (gemini-2.5-flash-native-audio-*) with affective dialogue — at some cost to tool-calling reliability.
• Reasoning-heavy → gpt-realtime-2.
• Translation → the dedicated translate models (their own protocol).

Selecting a provider per session

Applications typically read the provider from a request and build the matching model. The audio rates differ, so expose them too:

#[derive(Clone, Copy)]
enum Provider { OpenAI, Gemini }

impl Provider {
    fn audio_rates(self) -> (u32, u32) {   // (input, output)
        match self {
            Provider::OpenAI => (24_000, 24_000),
            Provider::Gemini => (16_000, 24_000),
        }
    }
}

fn build_model(p: Provider) -> anyhow::Result<(BoxedModel, &'static str)> {
    Ok(match p {
        Provider::OpenAI => (
            Arc::new(OpenAIRealtimeModel::new(std::env::var("OPENAI_API_KEY")?, "gpt-realtime")),
            "marin",
        ),
        Provider::Gemini => (
            Arc::new(GeminiRealtimeModel::new(
                GeminiLiveBackend::studio(std::env::var("GEMINI_API_KEY")?),
                "models/gemini-3.1-flash-live-preview",
            )),
            "Kore",
        ),
    })
}

The examples make all of these overridable with env vars (OPENAI_REALTIME_MODEL, GEMINI_REALTIME_MODEL) so you can pin a model without recompiling.

Capability matrix

Next: Tools →

Diagnostics and payload privacy

Realtime frames carry transcripts, tool arguments, tool results, and identifiers. When a recognized event fails to deserialize — provider schema drift — the warning reports a field-safe summary and withholds the frame:

Note: the digest groups log lines within a run. It is not a cryptographic digest and is not stable across processes.

To diagnose drift with the frame in hand, compile adk-realtime with the record-payloads feature, which records the first 300 bytes:

[dependencies]
adk-realtime = { version = "2.0.0", features = ["openai", "record-payloads"] }

The feature is off by default and is a deliberate choice per build, because schema drift is exactly when operators widen log collection and retention.