Video Connector for Node.js

The Vonage Video Connector Node.js library enables you to programmatically participate in Vonage Video API sessions as a server-side participant. This library allows you to connect to video sessions, publish and subscribe to streams, and process real-time audio and video data.

Important The Vonage Video Connector Node.js library is designed for server-side applications and requires valid Vonage Video API credentials and tokens with appropriate permissions.

This page covers the Node.js API. For the concepts, media formats, and runtime behavior shared by all Video Connector libraries, see the Video Connector guide. If you are working in Python, see Video Connector for Python.

This topic includes the following sections:

Getting started

The Vonage Video Connector for Node.js is distributed as the @vonage/video-connector package.

To install the library, run:

npm install @vonage/video-connector

The package ships prebuilt native binaries and requires no compilation step. TypeScript type declarations are included, so no separate @types package is needed.

Requirements

This library requires Node.js 18 or later running on Linux x64 or ARM64 platforms. We recommend using Debian Bookworm as it is the distribution where this has been most thoroughly tested.

Loading the library on an unsupported platform throws at import time, because the native module for that platform and architecture combination does not exist.

The promise-based API

The Node.js library exposes lifecycle operations as promises rather than completion callbacks. This shapes the whole API, so it is worth understanding before reading the rest of this page.

Each of connect(), publish(), subscribe(), disconnect(), unpublish(), and unsubscribe() returns a promise:

Method Resolves with Resolves when
connect() Session The session connection is established
publish() Publisher Your published stream has been created
subscribe() Subscriber The subscription is established
disconnect() void Disconnection completes
unpublish() void Unpublishing completes
unsubscribe() void Unsubscription completes

As a consequence, the one-shot lifecycle events have no corresponding callback in this library — the promise is the notification. There is no onConnected, no publisher onStreamCreated, and no subscriber onConnected to register. Every other event is a callback you pass in the arguments object.

Error handling follows the same split. If an error occurs before an operation settles, the promise rejects and your onError callback is not invoked; the rejection is the error notification. If an error occurs after the operation has settled, your onError callback is invoked as normal.

import { VonageVideoClient } from '@vonage/video-connector';

const client = new VonageVideoClient();

try {
  const session = await client.connect(applicationId, sessionId, token, {
    // Fires only for errors after the connection is established
    onError: (session, description, code) => {
      console.error(`Session error [${code}]: ${description}`);
    },
  });
  console.log(`Connected to session ${session.sessionId}`);
} catch (error) {
  // Failure to connect surfaces here, not in onError
  console.error('Failed to connect:', error);
}

Important Create only one VonageVideoClient instance per process. The underlying SDK supports a single active connection, and creating multiple instances results in undefined behavior.

Data structures

All types below are exported from the package and can be imported as types.

Session

Represents a Vonage Video API session that clients can connect to:

interface Session {
  sessionId: string;  // Unique identifier for the session
}

The Session object is passed to session-level callbacks to identify which session triggered the event.

Connection

Represents a participant's connection to a session:

interface Connection {
  connectionId: string;  // Unique identifier for the connection
  creationTime: number;  // Unix timestamp in milliseconds
  data: string;          // Connection data (encoded in the token)
}

Connection data can be used to store custom metadata about participants, such as user IDs or roles.

Note that creationTime is a numeric Unix timestamp in milliseconds, so it can be passed straight to new Date(connection.creationTime).

Stream

Represents a media stream (audio/video) published by a participant:

interface Stream {
  streamId: string;       // Unique identifier for the stream
  connection: Connection; // The connection that published this stream
}

Publisher

Represents your published stream in the session:

interface Publisher {
  stream: Stream;  // The underlying stream for this publisher
}

Subscriber

Represents a subscription to another participant's stream:

interface Subscriber {
  stream: Stream;  // The stream being subscribed to
}

AudioData

Represents audio data being transmitted or received:

interface AudioData {
  data: Buffer;                        // Raw PCM 16-bit LE samples
  sampleRate: SampleRate;              // 8000 | 12000 | 16000 | 24000 | 32000 | 44100 | 48000
  numberOfChannels: NumberOfChannels;  // 1 (mono) or 2 (stereo)
  numberOfFrames: number;              // One frame = one sample per channel
}

Audio format requirements:

  • data must be a Buffer containing 16-bit signed little-endian samples
  • Valid sample rates: 8000, 12000, 16000, 24000, 32000, 44100, 48000 Hz
  • Channels: 1 (mono) or 2 (stereo)
  • data.byteLength must be at least numberOfFrames * numberOfChannels * 2 bytes

VideoFrame

Represents video frame data being transmitted or received:

interface VideoFrame {
  data: Buffer;               // Raw pixel data
  width: number;              // Frame width in pixels
  height: number;             // Frame height in pixels
  format?: VideoFrameFormat;  // 'YUV420P' | 'RGB24' | 'ARGB32', defaults to 'YUV420P'
}

Video format requirements:

  • data must be a Buffer of 8-bit unsigned values
  • width and height are flat properties on the frame; there is no separate resolution object
  • Maximum resolution: 1920x1080 pixels (2,073,600 total pixels)
  • Required buffer size depends on the format:
Format Required data.byteLength
YUV420P width * height * 3 / 2
RGB24 width * height * 3
ARGB32 width * height * 4

VideoResolution

Represents the dimensions of a video frame:

interface VideoResolution {
  width: number;   // Width in pixels. Must be a positive integer
  height: number;  // Height in pixels. Must be a positive integer
}

Total pixel count (width * height) must not exceed 1920 * 1080 (2,073,600). This type is used by SubscriberVideoSettings.preferredResolution; the publisher geometry in SessionVideoPublisherSettings and on VideoFrame uses flat width and height properties instead.

CaptionsData

Represents caption text data received from a subscribed stream:

interface CaptionsData {
  text: string;     // The caption text content
  isFinal: boolean; // true for final captions, false for interim/partial captions
}

MediaBufferStats

Provides statistics about the publishing media buffers:

interface BufferStats {
  durationMs: number;  // Duration of queued media in milliseconds
}

interface MediaBufferStats {
  audio: BufferStats | null;  // null if no audio publisher is active
  video: BufferStats | null;  // null if no video publisher is active
}

Durations are plain numbers in milliseconds.

Validation types

The library exports union types for the values it validates, so invalid literals are caught at compile time:

type SampleRate = 8000 | 12000 | 16000 | 24000 | 32000 | 44100 | 48000;
type NumberOfChannels = 1 | 2;
type LogLevel = 'ERROR' | 'WARN' | 'INFO' | 'DEBUG' | 'TRACE';
type VideoFrameFormat = 'YUV420P' | 'RGB24' | 'ARGB32';

Configuration structures

SessionSettings

Configures session-level behavior:

import type { SessionSettings } from '@vonage/video-connector';

const sessionSettings: SessionSettings = {
  enableMigration: false,   // Enable automatic session migration. Defaults to false
  av: avSettings,           // Optional audio/video configuration
  logging: { level: 'INFO' } // Optional logging configuration
};

SessionAVSettings

Configures audio and video settings for the session:

import type { SessionAVSettings } from '@vonage/video-connector';

const avSettings: SessionAVSettings = {
  audioPublisher: { sampleRate: 48000, channels: 2 },
  audioSubscribersMix: { sampleRate: 48000, channels: 1 },
  videoPublisher: { width: 1280, height: 720, fps: 30, format: 'YUV420P' }
};
  • audioPublisher: Defines the format for audio data you provide via addAudio(). The audio data you send must match this configuration's sample rate and channel count.
  • audioSubscribersMix: Defines the format for the mixed audio you receive from all subscribed streams via the session-level onAudioData callback. The library automatically handles mixing multiple subscribers' audio and resampling/channel conversion to match your specified format.

For guidance on choosing these formats, see Audio for publishing versus subscribing.

SessionAudioSettings

Configures audio format for publishing or receiving audio data:

import type { SessionAudioSettings } from '@vonage/video-connector';

const audioSettings: SessionAudioSettings = {
  sampleRate: 48000,  // Optional, defaults to 48000
  channels: 1         // Optional, 1 (mono) or 2 (stereo), defaults to 2
};

Note that the property is named channels here, while AudioData uses numberOfChannels.

SessionVideoPublisherSettings

Configures video settings for publishing:

import type { SessionVideoPublisherSettings } from '@vonage/video-connector';

const videoSettings: SessionVideoPublisherSettings = {
  width: 1280,        // Optional, defaults to 640
  height: 720,        // Optional, defaults to 480
  fps: 30,            // Optional, 1-30, defaults to 30
  format: 'YUV420P'   // Optional, defaults to 'YUV420P'
};

Width and height are flat properties rather than a nested resolution object.

LoggingSettings

Controls logging verbosity:

import type { LoggingSettings } from '@vonage/video-connector';

const loggingSettings: LoggingSettings = {
  level: 'INFO'  // Required: 'ERROR' | 'WARN' | 'INFO' | 'DEBUG' | 'TRACE'
};

PublisherSettings

Configures your published stream:

import type { PublisherSettings } from '@vonage/video-connector';

const publisherSettings: PublisherSettings = {
  name: 'My Application',  // Required: name for your published stream
  hasAudio: true,          // Required: whether to publish audio
  hasVideo: true,          // Required: whether to publish video
  enableCaptions: false,   // Optional: enable live captions for this stream. Defaults to false
  audioSettings: {         // Optional
    enableStereoMode: true,
    enableOpusDtx: false
  }
};

Note: At least one of hasAudio or hasVideo must be true. Passing both as false throws a ValidationError synchronously from publish().

Note: Set enableCaptions: true to allow subscribers to receive live caption text from this stream via the subscriber onCaptionText callback. Captions are disabled by default.

PublisherAudioSettings

Configures audio settings for your published stream:

import type { PublisherAudioSettings } from '@vonage/video-connector';

const audioSettings: PublisherAudioSettings = {
  enableStereoMode: true,  // Optional: publish in stereo. Defaults to true
  enableOpusDtx: false     // Optional: enable discontinuous transmission. Defaults to false
};

Discontinuous transmission (DTX) stops sending audio packets during silence, saving bandwidth.

SubscriberSettings

Configures subscriber behavior:

import type { SubscriberSettings } from '@vonage/video-connector';

const subscriberSettings: SubscriberSettings = {
  subscribeToAudio: true,     // Optional, defaults to true
  subscribeToVideo: true,     // Optional, defaults to true
  subscribeToCaptions: false, // Optional, defaults to false
  videoSettings: {
    preferredResolution: { width: 640, height: 480 },
    preferredFrameRate: 15
  }
};

Captions must be enabled on the publishing side and opted into per subscriber via subscribeToCaptions.

SubscriberVideoSettings

Configures video preferences for subscribers:

import type { SubscriberVideoSettings } from '@vonage/video-connector';

const videoSettings: SubscriberVideoSettings = {
  preferredResolution: { width: 640, height: 480 },  // Optional
  preferredFrameRate: 15                             // Optional, 1-30
};

These settings request a specific simulcast quality layer from the Vonage Video API SFU. For an explanation of how the SFU honors them, see Preferred subscriber resolution and frame rate.

Connecting to a session

Basic connection

To connect to a Vonage Video API session, you need your application ID (API key if using Tokbox), session ID, and a valid token:

import { VonageVideoClient, type SessionSettings } from '@vonage/video-connector';

const client = new VonageVideoClient();

const sessionSettings: SessionSettings = {
  enableMigration: false,
  av: {
    audioSubscribersMix: { sampleRate: 48000, channels: 1 }
  },
  logging: { level: 'INFO' }
};

const session = await client.connect(
  'your_application_id',
  'your_session_id',
  'your_token',
  { sessionSettings }
);

console.log(`Connected to session: ${session.sessionId}`);

Connection with all callbacks

For full session management, register all available ongoing callbacks:

const session = await client.connect(
  'your_application_id',
  'your_session_id',
  'your_token',
  {
    sessionSettings,
    onError: onSessionError,
    onDisconnected: onSessionDisconnected,
    onConnectionCreated: onConnectionCreated,
    onConnectionDropped: onConnectionDropped,
    onStreamReceived: onStreamReceived,
    onStreamDropped: onStreamDropped,
    onAudioData: onAudioData,
    onReadyForAudio: onReadyForAudio,
    onMediaBufferDrained: onMediaBufferDrained
  }
);

There is no onConnected callback — the awaited promise serves that purpose.

Disconnecting from a session

Disconnect from the session when done:

await client.disconnect();

Checking connection state

if (client.isConnected()) {
  console.log('Still connected');
}

Session settings

Audio and video configuration

Configure audio and video settings for the session to control the format of media data:

import type { SessionAVSettings } from '@vonage/video-connector';

const avSettings: SessionAVSettings = {
  // Format of the audio you supply via addAudio()
  audioPublisher: {
    sampleRate: 48000,  // Valid: 8000, 12000, 16000, 24000, 32000, 44100, 48000
    channels: 2         // 1 for mono, 2 for stereo
  },
  // Format of the mixed audio delivered to your session onAudioData callback
  audioSubscribersMix: {
    sampleRate: 48000,
    channels: 1
  },
  // Geometry and format of the video you supply via addVideo()
  videoPublisher: {
    width: 1280,
    height: 720,
    fps: 30,
    format: 'YUV420P'  // Valid: YUV420P, RGB24, ARGB32
  }
};

await client.connect(applicationId, sessionId, token, {
  sessionSettings: { av: avSettings }
});

Invalid values in these settings throw a TypeError synchronously from connect(), before any network activity.

Logging configuration

Control the verbosity of console logging:

await client.connect(applicationId, sessionId, token, {
  sessionSettings: {
    logging: { level: 'DEBUG' }  // Valid: ERROR, WARN, INFO, DEBUG, TRACE
  }
});

Session migration

Enable automatic session migration in case of SFU rotation:

await client.connect(applicationId, sessionId, token, {
  sessionSettings: {
    enableMigration: true,
    av: avSettings,
    logging: { level: 'INFO' }
  }
});

Publishing streams

Publisher configuration

Configure publisher settings before starting to publish:

import type { PublisherSettings } from '@vonage/video-connector';

// Audio and video publisher
const publisherSettings: PublisherSettings = {
  name: 'AI Assistant Bot',
  hasAudio: true,
  hasVideo: true,
  enableCaptions: true,
  audioSettings: {
    enableStereoMode: true,
    enableOpusDtx: false
  }
};

// Or audio-only publisher
const audioOnlySettings: PublisherSettings = {
  name: 'Audio Bot',
  hasAudio: true,
  hasVideo: false,
  enableCaptions: false
};

Start publishing

Begin publishing a stream to the session:

const publisher = await client.publish({
  settings: publisherSettings,
  onError: onPublisherError,
  onStreamDestroyed: onStreamDestroyed
});

console.log(`Publishing stream: ${publisher.stream.streamId}`);

The promise resolves once the stream has been created, so there is no onStreamCreated callback. The onStreamDestroyed callback fires only when the stream is destroyed unexpectedly, such as a server-side forced unpublish — it does not fire when you call unpublish() yourself.

Important If you're publishing audio (hasAudio: true), you must wait for the onReadyForAudio callback before calling addAudio(). This callback indicates that the audio system is initialized and ready to accept audio data. Frames submitted beforehand are silently dropped by the native layer. This requirement does not apply to video-only publishing scenarios.

// onReadyForAudio is registered at connect() time, so capture its resolver first
let signalAudioReady: () => void = () => {};
const audioReady = new Promise<void>((resolve) => {
  signalAudioReady = resolve;
});

await client.connect(applicationId, sessionId, token, {
  sessionSettings,
  onReadyForAudio: () => {
    console.log('Audio system ready - can now add audio');
    signalAudioReady();
  }
});

await client.publish({ settings: publisherSettings });
await audioReady;

// Now safe to add audio
client.addAudio(audioData);

Video has no equivalent readiness signal: frames can be submitted as soon as publish() resolves.

Adding audio data

Send audio data to your published stream:

import type { AudioData } from '@vonage/video-connector';

const sampleRate = 48000;
const numberOfChannels = 1;
const numberOfFrames = 960;  // 20ms at 48kHz

// PCM16 LE: 2 bytes per sample per channel
const data = Buffer.alloc(numberOfFrames * numberOfChannels * 2);

const audioData: AudioData = {
  data,
  sampleRate,
  numberOfChannels,
  numberOfFrames
};

const accepted = client.addAudio(audioData);

addAudio() is synchronous. It returns true if the frame was accepted, and false if the controller is not initialized or the native call fails. It throws a TypeError if the frame is malformed — for example if the buffer is too small for the declared frame geometry.

Stop publishing

Stop publishing when done:

await client.unpublish();

Checking publishing state

if (client.isPublishing()) {
  console.log('Still publishing');
}

Subscribing to streams

Subscribe to streams

When a new stream is received, subscribe to it to receive audio and/or video data:

import type { Session, Stream } from '@vonage/video-connector';

async function onStreamReceived(session: Session, stream: Stream): Promise<void> {
  console.log(`New stream received: ${stream.streamId}`);
  console.log(`From connection: ${stream.connection.connectionId}`);

  try {
    const subscriber = await client.subscribe(stream, {
      settings: {
        subscribeToAudio: true,
        subscribeToVideo: true,
        subscribeToCaptions: true,
        videoSettings: {
          preferredResolution: { width: 640, height: 480 },
          preferredFrameRate: 15
        }
      },
      onError: onSubscriberError,
      onDisconnected: onSubscriberDisconnected,
      onRenderFrame: onRenderFrame,
      onAudioData: onSubscriberAudioData,
      onCaptionText: onCaptionText
    });

    console.log(`Subscribed to stream: ${subscriber.stream.streamId}`);
  } catch (error) {
    // onStreamReceived is a void callback, so handle failures here rather than
    // letting them become an unhandled rejection.
    console.error(`Failed to subscribe to ${stream.streamId}:`, error);
  }
}

Receiving subscribed media

The library delivers audio and video through different callbacks. For the reasoning behind this design, see Subscribing to streams in the Video Connector guide.

Important The Buffer inside an AudioData or VideoFrame delivered to a callback is only valid for the duration of that callback. If you need to retain the media beyond the callback — to queue it, or process it asynchronously — copy it first, for example with Buffer.from(frame.data).

Video data: Video frames are delivered individually per subscribed stream through the onRenderFrame callback registered at subscription time. Each invocation includes the subscriber that identifies which stream the frame belongs to, so you can process video from different participants separately.

import type { Subscriber, VideoFrame } from '@vonage/video-connector';

function onRenderFrame(subscriber: Subscriber, frame: VideoFrame): void {
  const streamId = subscriber.stream.streamId;
  console.log(`Video frame from stream ${streamId}: ${frame.width}x${frame.height}`);
  // Copy before retaining beyond this callback
  const owned = Buffer.from(frame.data);
}

Audio data: Mixed audio is delivered through the onAudioData callback registered during connect(), approximately every 10 ms. The library automatically mixes audio from all subscribed streams into a single stream. You cannot distinguish between individual participants' audio in this callback.

import type { Session, AudioData } from '@vonage/video-connector';

function onAudioData(session: Session, audioData: AudioData): void {
  console.log(
    `Mixed audio from all subscribers: ${audioData.sampleRate}Hz, ` +
    `${audioData.numberOfChannels} channel(s)`
  );
}

Caption data: This feature is currently available as a beta feature. Caption text is delivered per subscribed stream through the onCaptionText callback, with the subscriber identifying the source stream and a CaptionsData object containing the text and whether the result is final or interim.

Note For the onCaptionText callback to receive caption data, live captions must be enabled in the underlying Vonage Video API session configuration (outside of this library; see the Vonage Video API Live Captions documentation) and for the specific publisher stream that is sending audio. Set subscribeToCaptions: true in the subscriber settings to receive them.

import type { Subscriber, CaptionsData } from '@vonage/video-connector';

function onCaptionText(subscriber: Subscriber, captions: CaptionsData): void {
  const status = captions.isFinal ? 'final' : 'interim';
  console.log(`Caption (${status}) from stream ${subscriber.stream.streamId}: ${captions.text}`);
}

Individual audio data

This capability is currently available as a beta feature. Individual stream audio is delivered via the onAudioData callback registered at subscription time through subscribe() — distinct from the session-level onAudioData, which carries the mix. Audio arrives in the format received from the stream, Linear PCM 16-bit, and neither the sample rate nor the channel count can be configured prior to reception.

import type { Subscriber, AudioData } from '@vonage/video-connector';

function onSubscriberAudioData(subscriber: Subscriber, audioData: AudioData): void {
  console.log(
    `Individual audio from stream ${subscriber.stream.streamId}: ` +
    `${audioData.sampleRate}Hz, ${audioData.numberOfChannels} channel(s)`
  );
}

Unsubscribe from streams

Stop receiving media from a specific stream:

import type { Session, Stream } from '@vonage/video-connector';

function onStreamDropped(session: Session, stream: Stream): void {
  console.log(`Stream dropped: ${stream.streamId}`);
  client.unsubscribe(stream).catch((error) => {
    console.error('Failed to unsubscribe:', error);
  });
}

Audio data handling

Audio format

Audio data is exchanged as Linear PCM 16-bit signed little-endian samples in a Buffer:

  • Sample rates: 8000, 12000, 16000, 24000, 32000, 44100, or 48000 Hz
  • Channels: 1 (mono) or 2 (stereo)
  • Format: 16-bit signed little-endian in a Buffer
  • Frame size: Typically 20ms chunks (varies by sample rate)

Processing audio data

Handle incoming audio in the session audio data callback:

import type { Session, AudioData } from '@vonage/video-connector';

function onAudioData(session: Session, audioData: AudioData): void {
  const { sampleRate, numberOfChannels, numberOfFrames, data } = audioData;

  console.log(`Received ${numberOfFrames} frames at ${sampleRate}Hz, ${numberOfChannels} channel(s)`);

  // Read individual samples as 16-bit signed little-endian
  const firstSample = data.readInt16LE(0);

  // Copy if the audio is needed after this callback returns
  const owned = Buffer.from(data);
}

Creating audio data

When adding audio, build a Buffer of 16-bit samples:

const sampleRate = 48000;
const durationMs = 20;
const numberOfChannels = 1;
const numberOfFrames = (sampleRate * durationMs) / 1000;  // 960 frames

const data = Buffer.alloc(numberOfFrames * numberOfChannels * 2);

// Example: generate a 440Hz sine wave
for (let i = 0; i < numberOfFrames; i++) {
  const sample = Math.round(32767 * 0.5 * Math.sin((2 * Math.PI * 440 * i) / sampleRate));
  data.writeInt16LE(sample, i * 2);
}

client.addAudio({ data, sampleRate, numberOfChannels: 1, numberOfFrames });

Audio data continuity

When you publish audio, the library sends silence until your first addAudio() call, tolerates brief gaps without sending packets, then falls back to explicit silence frames, and pads partial periods to prevent drift. See Audio continuity for the full behavior.

Best practices:

  • Maintain a consistent audio rate by calling addAudio() at regular intervals matching your configured sample rate
  • Monitor buffer statistics using getMediaBufferStats() to ensure adequate audio data
  • Handle the onMediaBufferDrained callback to detect when your audio buffer is depleted
  • Consider implementing an audio generation strategy that adapts to varying processing loads

Video data handling

Video format

Video data is exchanged as 8-bit unsigned values in a Buffer, in one of three formats:

  • YUV420P: Planar YUV format with 4:2:0 chroma subsampling
  • RGB24: Packed RGB, 8 bits per channel
  • ARGB32: Packed ARGB, 8 bits per channel including alpha

Video specifications:

  • Resolutions: Up to 1920x1080 (Full HD)
  • Frame rates: 1-30 FPS
  • Format: 8-bit unsigned values in a Buffer

Processing video frames

Handle incoming video frames in the render frame callback:

import type { Subscriber, VideoFrame } from '@vonage/video-connector';

function onRenderFrame(subscriber: Subscriber, frame: VideoFrame): void {
  const { width, height, format, data } = frame;

  console.log(`Received ${width}x${height} frame in ${format} format`);

  // Copy if the frame is needed after this callback returns
  const owned = Buffer.from(data);
}

Creating video frames

When publishing video, build a Buffer sized for the format and resolution:

import type { VideoFrame } from '@vonage/video-connector';

const width = 640;
const height = 480;

// YUV420P layout:
// Y plane: width * height
// U plane: (width / 2) * (height / 2)
// V plane: (width / 2) * (height / 2)
const ySize = width * height;
const uvSize = (width / 2) * (height / 2);
const data = Buffer.alloc(ySize + 2 * uvSize);

// Fill with mid-gray
data.fill(128);

const videoFrame: VideoFrame = { data, width, height, format: 'YUV420P' };

client.addVideo(videoFrame);

addVideo() is synchronous. It returns true if the frame was accepted, and false if the controller is not initialized or the native call fails. It throws a TypeError if the frame is malformed — an unknown format, a non-positive dimension, a pixel count above 1920x1080, or a buffer too small for the declared resolution.

Video frame continuity

When you publish video, the library sends black frames until your first addVideo() call, repeats your last frame for up to 2 seconds if you stop supplying frames, then falls back to black frames. See Video continuity for the full behavior.

Best practices:

  • Maintain a consistent frame rate by calling addVideo() at regular intervals matching your configured FPS
  • Monitor buffer statistics using getMediaBufferStats() to ensure adequate video data
  • Handle the onMediaBufferDrained callback to detect when your video buffer is depleted
  • Consider implementing a frame generation strategy that adapts to varying processing loads

Media buffer management

Checking buffer stats

Monitor the state of your media buffers:

const stats = client.getMediaBufferStats();

if (stats.audio) {
  console.log(`Audio buffer duration: ${stats.audio.durationMs}ms`);
}

if (stats.video) {
  console.log(`Video buffer duration: ${stats.video.durationMs}ms`);
}

Each field is null when no publisher of that media type is active.

Clearing media buffers

Clear both audio and video buffers when needed, for example to interrupt playback mid-utterance:

if (client.clearMediaBuffers()) {
  console.log('Media buffers cleared successfully');
}

Buffer drained callback

Handle buffer drain events:

import type { MediaBufferStats } from '@vonage/video-connector';

function onMediaBufferDrained(stats: MediaBufferStats): void {
  console.log('Media buffers drained');

  if (stats.audio) {
    console.log(`Audio buffer: ${stats.audio.durationMs}ms remaining`);
  }

  if (stats.video) {
    console.log(`Video buffer: ${stats.video.durationMs}ms remaining`);
  }
}

The onMediaBufferDrained callback is registered in connect(). It is invoked when the internal audio or video buffers are depleted, and implements hysteresis so it does not fire repeatedly while the buffer remains empty. See Buffer drain events for details.

Getting connection info

Retrieve your local connection information:

const connection = client.getConnection();

if (connection) {
  console.log(`Connection ID: ${connection.connectionId}`);
  console.log(`Connection data: ${connection.data}`);
  console.log(`Created at: ${new Date(connection.creationTime).toISOString()}`);
}

getConnection() returns null when the client is not connected.

Event callbacks

Callbacks are grouped by the call that registers them. Remember that the one-shot lifecycle events — session connected, publisher stream created, and subscriber connected — are delivered by the promises returned from connect(), publish(), and subscribe() rather than as callbacks.

Session callbacks

Registered in connect():

Callback Signature
onError (session: Session, description: string, code: number) => void
onDisconnected (session: Session) => void
onConnectionCreated (session: Session, connection: Connection) => void
onConnectionDropped (session: Session, connection: Connection) => void
onStreamReceived (session: Session, stream: Stream) => void
onStreamDropped (session: Session, stream: Stream) => void
onAudioData (session: Session, audioData: AudioData) => void
onReadyForAudio (session: Session) => void
onMediaBufferDrained (stats: MediaBufferStats) => void 
import type { Session, Connection, Stream } from '@vonage/video-connector';

function onSessionError(session: Session, description: string, code: number): void {
  console.error(`Session error: ${description} (Code: ${code})`);
}

function onSessionDisconnected(session: Session): void {
  console.log(`Disconnected from session: ${session.sessionId}`);
}

function onConnectionCreated(session: Session, connection: Connection): void {
  console.log(`Participant joined: ${connection.connectionId}`);
  console.log(`Connection data: ${connection.data}`);
}

function onConnectionDropped(session: Session, connection: Connection): void {
  console.log(`Participant left: ${connection.connectionId}`);
}

function onStreamReceived(session: Session, stream: Stream): void {
  console.log(`Stream received: ${stream.streamId} from ${stream.connection.connectionId}`);
  // Decide whether to subscribe based on your application logic
}

Publisher callbacks

Registered in publish():

Callback Signature
onError (publisher: Publisher, description: string, code: number) => void
onStreamDestroyed (publisher: Publisher) => void 
import type { Publisher } from '@vonage/video-connector';

function onPublisherError(publisher: Publisher, description: string, code: number): void {
  console.error(`Publisher error: ${description} (Code: ${code})`);
}

function onStreamDestroyed(publisher: Publisher): void {
  console.log(`Published stream destroyed: ${publisher.stream.streamId}`);
}

Subscriber callbacks

Registered in subscribe():

Callback Signature
onError (subscriber: Subscriber, description: string, code: number) => void
onDisconnected (subscriber: Subscriber) => void
onRenderFrame (subscriber: Subscriber, frame: VideoFrame) => void
onAudioData (subscriber: Subscriber, audioData: AudioData) => void
onCaptionText (subscriber: Subscriber, captions: CaptionsData) => void 
import type { Subscriber } from '@vonage/video-connector';

function onSubscriberError(subscriber: Subscriber, description: string, code: number): void {
  console.error(`Subscriber error: ${description} (Code: ${code})`);
}

function onSubscriberDisconnected(subscriber: Subscriber): void {
  console.log(`Unsubscribed from stream: ${subscriber.stream.streamId}`);
}

Error handling

The library reports problems in three distinct ways.

Synchronous throws for invalid input. Arguments are validated before crossing into the native layer. addAudio(), addVideo(), and the audio/video settings passed to connect() throw a TypeError when a value has the wrong type or violates a documented constraint. publish() throws a ValidationError when the settings are well-typed but break a rule, such as hasAudio and hasVideo both being false.

import { ValidationError } from '@vonage/video-connector';

try {
  await client.publish({ settings: { name: 'Bot', hasAudio: false, hasVideo: false } });
} catch (error) {
  if (error instanceof ValidationError) {
    console.error('Invalid publisher settings:', error.message);
  }
}

ValidationError is a class exported by the package, so it can be caught with instanceof.

Promise rejections for failed operations. If an operation fails before it settles, its promise rejects with an Error whose message includes the underlying description and code.

try {
  await client.connect(applicationId, sessionId, token, { sessionSettings });
} catch (error) {
  // e.g. "Connection error [1004]: Authentication error"
  // or "Failed to initiate connection. Client may already be connected."
  console.error(error);
}

Callbacks for errors after settlement. Once an operation has succeeded, later failures are reported to the onError callback registered for that scope. An onError callback is therefore never invoked for a failure that already rejected the promise.

Boolean returns for media submission. addAudio(), addVideo(), and clearMediaBuffers() return false rather than throwing when the controller is not initialized or the native call fails.

Resource cleanup

Always clean up resources properly:

try {
  await client.connect(applicationId, sessionId, token, { sessionSettings });
  await client.publish({ settings: publisherSettings });
  // ... do work ...
} catch (error) {
  console.error('Application error:', error);
} finally {
  if (client.isPublishing()) {
    await client.unpublish().catch(() => console.warn('Failed to stop publishing'));
  }

  if (client.isConnected()) {
    await client.disconnect().catch(() => console.warn('Failed to disconnect'));
  }
}