Wiro AI Realtime Voice

--- name: wiro-realtime-voice description: Build realtime voice features with Wiro AI, such as live voice conversation, streaming text-to-speech and streaming speech-to-text, by starting a task through the REST API and streaming PCM audio over the Wiro WebSocket. Use when the user wants a voice assistant, a live voice conversation, streaming text-to-speech or speech-to-text, or WebSocket audio in an app built on Wiro. --- # Wiro Realtime Voice A Wiro realtime model keeps a session open. One REST call starts a task, then raw PCM audio and JSON events flow over a WebSocket until the session ends. For the REST basics (auth headers, model search, prices, task lookup), use the `wiro-ai` skill. ## When to use it | Mode | Client sends | Server sends | Good for | | --- | --- | --- | --- | | Voice conversation | Microphone audio | AI audio, plus `TRANSCRIPT_USER:` and `TRANSCRIPT_AI:` text | Voice assistants, interactive voice chat | | Text-to-speech | Text in the Run call, no microphone | AI audio only | Narration, voiceover, assistants | | Speech-to-text | Microphone audio | `TRANSCRIPT_USER:` text only, no audio | Live dictation, captioning, meeting transcription | For a finished audio file rather than a live stream, a standard text-to-speech model through Run fits better (see `wiro-ai`). ## Start a session 1. Pick a model and read its parameters with `POST /Tool/Detail`. They differ per model. 2. From server-side code (see Credentials in apps), create the task with `POST https://api.wiro.ai/v1/Run/{owner}/{model}`. Use this asynchronous Run; the `/sync` variant is for finite models. 3. Open `wss://socket.wiro.ai/v1` and register with `task_info`. 4. Wait for `task_stream_ready`. Send no audio before it. 5. Stream microphone frames, play the audio frames that come back, or both. 6. Send `task_session_end`, wait for `task_postprocess_end`, then close the socket. Run body for `openai/gpt-realtime-mini` from the docs, sent with `x-api-key: YOUR_API_KEY`: ```json { "voice": "marin", "system_instructions": "You are a friendly assistant.", "input_audio_format": "audio/pcm", "output_audio_format": "audio/pcm", "input_audio_rate": "24000", "output_audio_rate": "24000" } ``` The other docs examples send `{"text": "Hello, this is a realtime text-to-speech demo."}` to `openmoss/moss-tts-realtime` and `{"transcriptionDelay": "balanced"}` to `mistralai/voxtral-mini-4b-realtime-2602`. Run returns the task id and the task token. Check `result` and `errors` before you connect: ```json {"result": true, "errors": [], "taskid": "TASK_ID", "socketaccesstoken": "TASK_TOKEN"} ``` The client sends these JSON text messages, plus binary audio frames (see Audio): ```json {"type": "task_info", "tasktoken": "TASK_TOKEN"} {"type": "task_session_interrupt", "tasktoken": "TASK_TOKEN"} {"type": "task_session_end", "tasktoken": "TASK_TOKEN"} ``` The server sends: | Event | Meaning | | --- | --- | | `task_stream_ready` | The session is ready. Start the microphone. In TTS, audio starts arriving. | | Binary frame | AI audio (conversation and TTS). | | `task_output` | A transcript in `message`, prefixed `TRANSCRIPT_USER:` (the person) or `TRANSCRIPT_AI:` (the AI). | | `task_stream_end` | Conversation: the AI finished its turn. TTS: audio for this segment is done. STT: no more transcripts will come. | | `task_cost` | `turnCost`, `cumulativeCost` and `usage`, the provider's raw breakdown (its shape varies). | | `task_end` | The model process exited. Keep listening. | | `task_postprocess_end` | The last event. Close the socket now. | | `task_cancel` | The task was cancelled or killed. Treat it as the end. | ```json {"type": "task_output", "message": "TRANSCRIPT_USER:What's the weather like today?"} ``` The socket can also send the general task events from https://wiro.ai/docs/websocket (`task_queue`, `task_start` and so on); the examples treat them as information only. Strip the transcript prefix and update the UI on every message; speech-to-text sends partial results as words are recognized. - **Conversation turns:** keep the microphone streaming. To cut the AI off, stop local playback at once and send `task_session_interrupt`. The server stops that reply, sends a final `task_stream_end` for it, and the session goes on. Models whose provider does its own barge-in (the OpenAI realtime family and ElevenLabs) may also stop on their own when the person starts talking. The example apps rely on that with `gpt-realtime-mini` and silence their local playback. - **Text-to-speech:** the stream usually ends by itself once the text is spoken (`task_stream_end`, `task_cost`, `task_end`, `task_postprocess_end`). Let queued audio finish before you close the player. Send `task_session_end` to stop early. - **Speech-to-text:** keep sending audio until you end the session. The model returns text only when it hears speech. ## Audio - Both directions use raw PCM: signed 16-bit (Int16), 24,000 Hz, mono, little-endian. If the model takes format or rate parameters, send the values above (`audio/pcm`, `24000`) so the Run call matches your pipeline. For speech-to-text, always send 24 kHz; the server converts it for the model. - A binary frame is the task token as UTF-8 bytes, one `|` byte (0x7C), then the PCM bytes. Send audio only as binary frames. - On receive, check for a binary frame before you parse JSON, then take everything after the first `|` byte as PCM. - Chunks: the docs use 200 ms (4,800 samples, 9,600 bytes). The mobile examples send 20 ms (480 samples) so barge-in reacts faster. - Playback: convert Int16 to Float32 (divide by 32768), schedule each chunk at `max(currentTime, nextPlayTime)`, then advance `nextPlayTime` by the chunk's length, so chunks play back to back without gaps or clicks. - Browser capture (docs sample): `getUserMedia` with `echoCancellation` and `noiseSuppression`, an `AudioContext` at 24,000 Hz, and an AudioWorklet that buffers samples and converts them to Int16. - Mobile (examples): use one audio graph for capture and playback so the OS echo canceller hears both sides. For conversation, the iOS examples set `AVAudioSession` to `.playAndRecord` with mode `.voiceChat`, and the Android examples record from `VOICE_COMMUNICATION` and add `AcousticEchoCanceler` when the device has one. On Android, play TTS-only audio on a media track; a voice-communication track without a call route plays from the earpiece. - Permissions: conversation and STT need the microphone, TTS does not. iOS needs `NSMicrophoneUsageDescription` plus a runtime request. Android needs `RECORD_AUDIO` (requested at runtime) and `INTERNET`. Test on a physical device; the examples note that simulator and emulator microphones are not a usable voice path. Framing an outgoing chunk, adapted from ReactNative-Realtime `src/realtime/binaryFrame.ts` (MIT): ```ts import { utf8ToBytes } from '@noble/hashes/utils.js'; const SEPARATOR = '|'.charCodeAt(0); // 0x7C function encode(taskToken: string, payload: Uint8Array): Uint8Array { const prefix = utf8ToBytes(taskToken); const frame = new Uint8Array(prefix.length + 1 + payload.length); frame.set(prefix, 0); frame[prefix.length] = SEPARATOR; frame.set(payload, prefix.length + 1); return frame; // send as a binary WebSocket message } ``` ## Choose a model and know the cost - Conversation models in the docs: `openai/gpt-realtime-mini`, `openai/gpt-realtime` and `elevenlabs/realtime-conversational-ai`. - Text-to-speech and speech-to-text: browse https://wiro.ai/models with the Realtime TTS or Realtime STT category. The docs use `openmoss/moss-tts-realtime` and `mistralai/voxtral-mini-4b-realtime-2602`; the example apps also offer `openbmb/voxcpm2` for speech. - Typical parameters: voice and system instructions for conversation; the text and an optional reference audio clip for voice cloning for TTS; a transcription delay that trades speed for accuracy, and sometimes a language, for STT. Confirm them with `POST /Tool/Detail` before you build the Run body. - Each model has its own price, shown on its model page and at https://wiro.ai/product/pricing (`wiro-ai` reads it from the API). A session is charged while it runs; each `task_cost` event gives `turnCost` for the latest turn and the running total in `cumulativeCost`. Money rules: - Before you start a session for the user, a test call included, tell them the model and the expected cost. - A session bills while it runs. End sessions the user no longer needs. Ask before long or open-ended ones, such as always-on listening, a session left open in the background, or test loops. - In apps, end the session when the user leaves the screen, and cap how long one can run. The example apps hang up when their screen is left, and the Flutter one also ends a session when its own timer runs out. ## Credentials in apps - Never ship a Wiro API key or secret in a browser or mobile app. Build config values (`Secrets.xcconfig`, `secrets.properties`, `--dart-define`, `.env`) end up inside the binary; the examples use them for local runs only. - The key is needed for one call: `POST /Run`. The WebSocket subscribe, the audio frames and the session end are authorized by the per-task token alone, so the key can stay on your server. - The pattern: your backend holds `WIRO_API_KEY` (plus `WIRO_API_SECRET` for a Signature project), decides who may start a session and for how long, calls Run, and returns only `taskid` and `socketaccesstoken` to the app. It also stops tasks for the app with `POST /Task/Cancel` (`taskid`, while queued) or `POST /Task/Kill` (`socketaccesstoken`, once running). - Each example app keeps those calls behind a `WiroApiClient` interface (`runModel`, `terminateTaskBestEffort`), so you can point it at your backend. The Swift README section "Shipping to users" shows how. - The task token expires with the task. Keep tokens and keys out of logs. - Run auth headers: `x-api-key`, or `x-api-key`, `x-nonce` and `x-signature` for a Signature project. The `wiro-ai` skill shows the signing. ## Errors and cleanup - End every session the same way: `task_session_end`, a bounded wait for `task_postprocess_end`, then close the socket. If a client disconnects without it, the server stops the session, but do not rely on that. - A Run response with `result: false` or entries in `errors` has no task token, so there is no session to open. The conversation examples retry network, server-unavailable and timeout failures with backoff, and never retry authentication errors or rejected requests. - If `task_stream_ready` does not arrive in time, close the socket and have the backend stop that task before you try again, so nothing keeps running. - If the socket drops mid-conversation, the examples first resubscribe to the same task (a new socket, the same `task_info` and token) within a small retry budget, and start a new task only if that fails. A new task is a fresh assistant with no memory of the conversation. - The server can end a session too, for example when the wallet balance runs out; you still get `task_cost` and `task_end`. Treat `task_postprocess_end` or `task_cancel` as the end whenever they arrive, and do not start a replacement on your own. - On every exit path (stop, error, leaving the screen), stop the microphone and release the audio session. The examples also guard the microphone so two features never hold it at once. ## Example apps | Repo | Platform | What it shows | | --- | --- | --- | | [Swift-Realtime](https://github.com/wiroai/Swift-Realtime) | iOS, Swift and SwiftUI | One `AVAudioEngine` graph; the README section "Shipping to users" sketches a backend client | | [Kotlin-Realtime](https://github.com/wiroai/Kotlin-Realtime) | Android, Kotlin and Jetpack Compose | One `AudioRecord` and `AudioTrack` graph, session lifecycle tests with fakes | | [Dart-Realtime](https://github.com/wiroai/Dart-Realtime) | Flutter on Android and iOS | A native echo-cancelling audio package and a session time limit | | [ReactNative-Realtime](https://github.com/wiroai/ReactNative-Realtime) | React Native and TypeScript on Android and iOS | The protocol in pure TypeScript, native audio behind one `WiroPcmAudio` TurboModule | Each app has the same three tabs over one protocol: conversation with barge-in (`openai/gpt-realtime-mini`), live transcription (`mistralai/voxtral-mini-4b-realtime-2602`) and streaming text-to-speech (`openmoss/moss-tts-realtime` or `openbmb/voxcpm2`). Their auth, networking, realtime and audio layers have no UI dependency, so you can lift them into your own app. To run one, you need a Wiro project and API key from https://wiro.ai/panel/project/new. ## More - General Wiro skill (REST API, MCP server, auth, prices, tasks): `npx skills add wiroai/Wiro-Plugin --skill wiro-ai` - Realtime voice conversation: https://wiro.ai/docs/realtime-voice-conversation - Realtime text-to-speech: https://wiro.ai/docs/realtime-text-to-speech - Realtime speech-to-text: https://wiro.ai/docs/realtime-speech-to-text - WebSocket events: https://wiro.ai/docs/websocket - Run a model: https://wiro.ai/docs/run-a-model - Cancel and kill tasks: https://wiro.ai/docs/tasks - Models: https://wiro.ai/models - Pricing: https://wiro.ai/product/pricing

存档时间线

版本索引永久保留；内容副本只保留最新 1 份，更早版本仅留索引与哈希（存档时间线的证据链不会因此断裂）。

下载存档内容副本