Video Connector

The Vonage Video Connector enables you to programmatically participate in Vonage Video API sessions as a server-side participant. It allows you to connect to video sessions, publish and subscribe to streams, and process real-time audio and video data.

The library handles WebRTC connectivity, media processing, and session management automatically, allowing you to focus on building your application logic. Audio is delivered as Linear PCM 16-bit data, and video is delivered as 8-bit frames in YUV420P, RGB24, or ARGB32 formats, all at configurable sample rates, resolutions, and channel configurations.

This page describes the concepts and behavior shared by all Video Connector libraries. For installation instructions, API signatures, and code samples, see the guide for your language.

Choose your library

Both libraries expose the same capabilities and are backed by the same native implementation. They differ in naming conventions and in how asynchronous operations are surfaced: the Python library uses completion callbacks, while the Node.js library returns promises. This page describes the behavior they share.

This topic includes the following sections:

• How it works
• Requirements
• Core concepts
• Media formats
• Session configuration
• Publishing media
• Subscribing to streams
• Media buffer management
• Event model
• Limits

How it works

The Video Connector joins a session as a regular WebRTC participant. From the perspective of other participants, it is indistinguishable from a browser or mobile client: it has its own connection, it can publish a stream, and it can subscribe to the streams of others.

Where it differs is that media is exchanged with your application code rather than with a camera, microphone, or screen. You push raw audio and video frames into the connector to publish them, and you receive raw audio and video out of the streams you subscribe to. This makes the connector suited to server-side workloads such as:

• Real-time voice and video AI agents
• Live transcription, translation, and captioning
• Recording, archiving, and compliance capture
• Computer vision and content moderation
• Audio and video effects processing

The typical lifecycle is:

1. Connect to a session using your application ID, session ID, and a token.
2. Publish a stream, then feed audio and/or video frames into it.
3. Subscribe to streams from other participants as they arrive, and process the media you receive.
4. Unsubscribe, unpublish, and disconnect when finished.

Requirements

The Video Connector ships as a native library with prebuilt binaries. It runs on Linux on x86_64 (AMD64) and ARM64 only.

We recommend Debian Bookworm, as it is the distribution where the connector has been most thoroughly tested.

Core concepts

The Video Connector uses a small set of objects to represent sessions, participants, streams, and media. Understanding these is essential for working with either library effectively.

Session

A Vonage Video API session that clients connect to. The session is identified by its ID and is passed to session-level event handlers so you can identify which session triggered an event.

Connection

A participant's connection to a session. Every participant, including the connector itself, has exactly one connection. A connection carries:

• A unique identifier
• A creation timestamp
• Connection data, which is encoded in the token used to connect

Connection data is useful for storing custom metadata about participants, such as user IDs or roles.

Stream

A media stream (audio, video, or both) published by a participant. A stream has a unique identifier and a reference to the connection that published it. Streams are announced to you as participants begin publishing, and are what you subscribe to in order to receive media.

Publisher

Your own published stream in the session. There is at most one publisher per connector instance. The publisher holds a reference to the stream it created, which is how other participants see you.

Subscriber

A subscription to another participant's stream. You create one subscriber per stream you want to receive media from, and each subscriber holds a reference to the stream it is subscribed to. Media and caption events are delivered with the subscriber that produced them, so you can tell which participant the data came from.

How they relate

Session
├── Connection (multiple participants)
│   └── Stream (participant's published media)
│       ├── Publisher (your published stream)
│       └── Subscriber (your subscription to their stream)
├── Audio data (flowing through streams)
└── Video frames (flowing through streams)

Media formats

Audio

Audio is always exchanged as Linear PCM, 16-bit signed integers. An audio frame is one sample per channel, so a buffer must hold at least (number of frames × number of channels) samples.

• Sample rates: 8000, 12000, 16000, 24000, 32000, 44100, or 48000 Hz
• Channels: 1 (mono) or 2 (stereo)
• Frame size: typically 20 ms chunks, varying by sample rate

Sample rate and channel count are configurable independently for the audio you publish and the mixed audio you receive. See Session configuration.

Video

Video is exchanged as 8-bit unsigned values in one of three pixel formats:

• Resolutions: up to 1920x1080 (2,073,600 pixels total)
• Frame rates: 1 to 30 FPS

Session configuration

Audio for publishing versus subscribing

The connector lets you configure two independent audio formats:

• Publisher audio defines the format of the audio data you provide when publishing. The audio you send must match this sample rate and channel count.
• Subscriber mix audio defines the format of the mixed audio you receive from all subscribed streams. The library handles mixing multiple participants and resampling or channel conversion to match your requested format.

This separation lets you optimize for your use case. For example:

• Publish in stereo for high-quality output while receiving a mono mix to simplify processing
• Publish at 16 kHz for speech while receiving at 48 kHz for high-fidelity playback
• Use different rates on each side to match the requirements of an audio processing pipeline

Preferred subscriber resolution and frame rate

When subscribing to routed streams that use simulcast, the Vonage Video API SFU (Selective Forwarding Unit) can send different quality layers of the video. Subscriber settings let you request a specific layer:

• Preferred resolution requests a spatial layer. The SFU sends the layer that most closely matches.
• Preferred frame rate requests a temporal layer. The SFU sends the layer that most closely matches.

These preferences help optimize bandwidth and processing on the subscriber side by requesting only the quality level you need, rather than always receiving the highest quality available.

Session migration

Session migration can be enabled so that the connector automatically migrates in the event of SFU rotation. It is disabled by default.

Logging

Console logging verbosity is configurable across five levels: ERROR, WARN, INFO, DEBUG, and TRACE.

Publishing media

A publisher must publish audio, video, or both. Configuring a publisher with neither is an error.

Waiting for audio readiness

Audio continuity

When you publish audio, the library maintains a continuous stream on your behalf:

Initial publishing. The library sends silence (zero-filled frames) until you provide your first audio data. This makes the stream immediately available to subscribers without waiting for your application to produce audio.

Silence tolerance. If you temporarily stop providing audio, the library tolerates brief gaps by not sending audio packets at all. This hysteresis prevents unnecessary silence packets during momentary processing delays.

Explicit silence. After the tolerance period, if no new audio is available, the library switches to sending explicit silence frames. This maintains the stream while indicating that no active audio is being provided.

Buffer flush. If you provide less than a full period's worth of audio, the library flushes the remaining data and pads it with silence to maintain correct timing and prevent audio drift.

Video continuity

When you publish video, the library maintains frame continuity on your behalf:

Initial publishing. The library sends black frames until you provide your first frame, so the stream is immediately available to subscribers.

Last frame repetition. If you stop providing frames, the library repeats the last frame you provided for up to 2 seconds, keeping playback smooth for subscribers.

Black frame fallback. After 2 seconds of repetition, the library switches to black frames. This signals to subscribers that video is no longer actively being provided while keeping the stream alive.

Best practices

• Submit media at regular intervals matching your configured sample rate and frame rate
• Monitor buffer statistics to confirm you are supplying enough data
• Handle the buffer-drained event to detect when your media buffers are depleted
• Adapt your media generation strategy to varying processing loads

Subscribing to streams

When a participant starts publishing, a stream-received event fires and you decide whether to subscribe. Media from your subscriptions is delivered through three separate channels.

Video is delivered per stream. Each frame arrives with the subscriber that identifies its source, so you can process each participant's video independently — for layout management, per-stream recording, or per-stream effects.

Mixed audio is delivered as a single stream at the session level. The library automatically mixes audio from all subscribed streams into one stream in the format you configured for the subscriber mix. Individual participants cannot be distinguished in this mixed audio.

Individual audio is delivered per stream, at the subscriber level. This capability is currently available as a beta feature. Audio arrives in the format received from the stream — Linear PCM 16-bit — and neither the sample rate nor the channel count can be configured before reception.

Captions are delivered per stream. This feature is currently available as a beta feature. Each caption event includes the subscriber identifying the source stream, the caption text, and whether the result is final or interim:

• Interim results are partial and may be updated as more speech is processed. Useful for live display.
• Final results are complete and will not change. Use these for storage or downstream processing.

Media buffer management

The connector maintains internal buffers for the audio and video you publish. You can inspect the amount of queued media at any time, and clear both buffers when you need to discard pending media — for example, when interrupting a bot mid-utterance.

Buffer drain events

A buffer-drained event fires when an internal audio or video buffer is depleted. This happens when media is transmitted to the session faster than your application supplies it. Treat the event as a signal to increase your media production rate or adjust your publishing strategy.

The event implements hysteresis to avoid excessive triggering: after an initial drain, it will not fire again until the buffer is replenished with new media and subsequently becomes depleted again. This prevents a flood of repeated notifications while the buffer remains empty.

Event model

Both libraries expose the same set of events, grouped by the object they belong to.

How these events are surfaced differs by language. In Python, every event is a callback you register. In Node.js, the one-shot lifecycle events — session connected, publisher stream created, and subscriber connected — are consumed by the promise returned from the corresponding method, and the remaining events are callbacks. See the language guide for details.

Limits