Using the Vonage Media Processor library

Use the Vonage Media Processor library to apply custom transformations to audio and video.

You can use the Publisher.setAudioMediaProcessorConnector(), Publisher.setVideoMediaProcessorConnector(), Subscriber.setAudioMediaProcessorConnector(), and Subscriber.setVideoMediaProcessorConnector(), methods to apply custom audio and video transformers that use the Vonage Media Processor library to audio and video. The Publisher methods will affect the subscribers' videos whereas the Subscriber methods will only apply to that single Subscriber.

OpenTok.js includes three ways to implement transformers:

  • Basic: Use the Publisher.applyVideoFilter() method to easily apply the pre-defined background blur or background replacement filter. Use the Publisher.applyAudioFilter() method to easily apply the pre-defined advanced noise suppression filter. These methods implement these filters using the Vonage Media Library, requiring only a few configuration settings:

See Applying a background blur or background replacement filter and Applying the advanced noise suppression filter.

  • Moderate: You can use transformers from theVonage ML transformers ([@vonage/ml-transformers](https://www.npmjs.com/package/@vonage/ml-transformers)) library.

    See the background image replacement and background blur sample apps.

  • Advanced: You can write your own transformer class that implements the Transformer class:

const transformer = new OverlayTextTransformer('Hello World'); const transformers = [ transformer ] const mediaProcessor = new MediaProcessor(); mediaProcessor.setTransformers(transformers); const mediaProcessorConnector = new MediaProcessorConnector(mediaProcessor); publisher.setVideoMediaProcessorConnector(mediaProcessorConnector);

See Applying a video transformer from the Vonage Media Library.

The Vonage Media Processor library, available at npm, facilitates the use of the Insertable Streams API to perform transform operations on video and audio tracks.

Applying a background blur or background replacement filter

See this documentation.

Applying the advanced noise suppression filter

See this documentation.

Applying a video transformer from the Vonage Media Library

The library includes two key classes:

  • MediaProcessorConnector — Accepts a MediaProcessor object in its constructor and handles input and output tracks. The OpenTok.js implementation handles the tracks. You only have to provide the MediaProcessorConnector instance.

To use a MediaProcessorConnector to connect a transformer to a publisher's audio or video, use the Publisher.setAudioMediaProcessorConnector() and Publisher.setVideoMediaProcessorConnector() methods.

Publisher.setVideoMediaProcessorConnector() method

The Publisher.setVideoMediaProcessorConnector() method applies a video transformer to a published stream:

Note: The Publisher.setVideoMediaProcessorConnector() method is incompatible with the Publisher.applyVideoFilter() method.

Publisher.setAudioMediaProcessorConnector() method

The Publisher.setAudioMediaProcessorConnector() applies an audio transformer to a published stream:

Subscriber.setVideoMediaProcessorConnector() method

The Subscriber.setVideoMediaProcessorConnector() method applies a video transformer to a subscribed stream:

Subscriber.setAudioMediaProcessorConnector() method

The Subscriber.setAudioMediaProcessorConnector() applies an audio transformer to a subscribed stream:

Client requirements

The Insertable Streams API used by the Publisher.setVideoMediaProcessorConnector(), Subscriber.setVideoMediaProcessorConnector() and Publisher.applyVideoFilter() methods are supported in recent versions of Chrome, Firefox, Electron, Opera, Edge, Samsung Internet, WebView Android and the latest versions of Safari (version 26+) and iOS WebView (version 26+). Publisher.setAudioMediaProcessorConnector(), Subscriber.setAudioMediaProcessorConnector() and Publisher.applyAudioFilter() are not supported in other (non-Chromium-based) browsers or on iOS. You can check if the client supports these features by calling the OT.hasMediaProcessorSupport() method.

Transformers, audio filters, and video filters require adequate processor support. Even in supported browsers, transformers may not be stable when background processes limit available processing resources. Use Chrome (desktop or Android) for optimal results.

For the best user experience, ensure laptop/desktop computers meet these minimum requirements:

  • CPU — Intel Core i5 (with 4 cores or higher) or Apple Silicon M1 (or higher)
  • 8G RAM (or more)
  • Dedicated GPU (for example, NVIDIA)

Some operating systems may offer multiple battery settings or modes to conserve energy (for example, to extend laptop battery life) by throttling CPU performance. This may result in suboptimal transformer performance and introduce unwanted audio or video artifacts. We recommend setting your operating system to use the best performance mode (or to not use low-power mode) in such cases.

Many video transformations (such as background blur) use segmentation to separate the speaker from the background. For best results, use proper lighting and a plain background. Insufficient lighting or complex backgrounds may cause video artifacts (for example, the speaker or a hat the speaker is wearing may get blurred along with the background).

You should perform benchmark tests on as many supported devices as possible, regardless of whether the transformation is a basic, moderate, or advanced implementation.

You can use the Session cpuPerformanceChanged event to detect CPU performance changes to disable video transformers as needed. See Detecting CPU performance changes.

For additional guidelines on optimizating call quality, see Call Quality Monitoring.

More about transformers and sample code

For a more information, see the Vonage Media Processor library documentation.

The opentok-web-samples repo includes the following sample apps using the Vonage Media Processor library to apply transformations to streams published in OpenTok.js:

  • Basic-Audio-Transformer — Shows how to apply an audio transformer using a web worker and a separate thread for the web worker.
  • Basic-Video-Transformer — Shows how to apply a video transformer using a canvas, a web worker, and a separate thread for the web worker.
  • Basic-Video-Zoom — Shows how to use a basic transformer to zoom the publisher.
  • Moderate-Background-Blur — Shows how to apply a video transformer from the @vonage/ml-transformers module.

There are additional samples that use the Vonage Media Transformers library here.

Other user interface customization options

For other available options for customizing the user interface, see the Customizing the UI guide.